IngestionPipeline: loading, splitting and embedding in one step

Splitting and embedding repeat whenever documents change. An ingestion pipeline names the steps once and, with a document store, skips unchanged documents.

Example

from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.storage.docstore import SimpleDocumentStore

pipeline = IngestionPipeline(
    transformations=[SentenceSplitter(chunk_size=80, chunk_overlap=0), Settings.embed_model],
    docstore=SimpleDocumentStore(),
)

transformations run in order on the documents: the splitter makes nodes, the embedding model adds a vector to each. The document store remembers which documents the pipeline has seen and what their content was.

Example

from llama_index.core.ingestion import IngestionPipeline
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.storage.docstore import SimpleDocumentStore

pipeline = IngestionPipeline(
    transformations=[SentenceSplitter(chunk_size=80, chunk_overlap=0), Settings.embed_model],
    docstore=SimpleDocumentStore(),
)

first = pipeline.run(documents=documents)
print("first run:", len(first), "nodes, embedded:", first[0].embedding is not None)

second = pipeline.run(documents=documents)
print("second run:", len(second), "nodes")

The second run returned nothing: every document was unchanged, so nothing was split or embedded again. With a hosted embedding model that is money saved on every re-run.

A changed document

Example

from llama_index.core import SimpleDirectoryReader

with open("help/delivery.md", "a") as f:
    f.write("\nOrders placed on a Sunday are sent on Monday.\n")

updated = SimpleDirectoryReader("help", file_metadata=only_name, filename_as_id=True).load_data()
print("after an edit:", len(pipeline.run(documents=updated)), "nodes")

filename_as_id=True gives each document an id based on its file name, so an edited file is recognised as the same document with new content. Only the changed file was processed again.

Try it yourself

Run the pipeline a third time after the edit.
Remove filename_as_id=True from the second reader and run it. What happens, and why?
Add a second splitter step with a smaller chunk size and count the nodes.

PreviousMetadata: what gets embedded and what the model sees

You understood something today that you didn't yesterday.