Indexing: from files to searchable documents

An indexing pipeline converts files to documents, cleans them, splits them into pieces a retriever can match precisely, and writes them to a store.

Two policy files, with the untidy spacing real files have:

Examplepolicies/delivery.txt

Parcels ship within two days.   They arrive by courier.


Tracking links are emailed when a parcel ships.

Four components, each from its own module: a converter, two preprocessors and a writer.

Example

from pathlib import Path

from haystack import Pipeline
from haystack.components.converters import TextFileToDocument
from haystack.components.preprocessors import DocumentCleaner, DocumentSplitter
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore

The pipeline runs them in that order, into an in-memory store:

Example

store = InMemoryDocumentStore()
indexing = Pipeline()
indexing.add_component("convert", TextFileToDocument())
indexing.add_component("clean", DocumentCleaner())
indexing.add_component("split", DocumentSplitter(split_by="period", split_length=1))
indexing.add_component("write", DocumentWriter(document_store=store))
indexing.connect("convert", "clean")
indexing.connect("clean", "split")
indexing.connect("split", "write")

Example

result = indexing.run({"convert": {"sources": sorted(Path("policies").glob("*.txt"))}})
print(result)
for document in store.filter_documents():
    print(repr(document.content), document.meta["file_path"])

TextFileToDocument reads each file into a document, with file_path in the metadata.
DocumentCleaner collapses repeated spaces and removes empty lines.
DocumentSplitter(split_by="period", split_length=1) makes one document per sentence, keeping the metadata and adding where the piece came from.
DocumentWriter writes them. connect("convert", "clean") without socket names works because each pair has one obvious match.

Five sentences became five documents. Splitting at a period keeps the space that followed it, which retrieval ignores. A retriever now matches the sentence about tracking links on its own, instead of a whole file where it is one line among several.

Sentence splitting

split_by="sentence" is smarter about abbreviations like "e.g.", but needs the nltk package and its tokenizer data. period, word, line and passage need nothing extra.

Try it yourself

Split by word with split_length=5 and split_overlap=2.
Print a document's full meta after splitting.
Run the pipeline twice and read the error, then set a policy on the writer.

PreviousDocuments and the document store

Little by little, you're building something great.