Documents and the document store

A Document holds content, metadata and an id. A document store keeps them; InMemoryDocumentStore needs no database and supports keyword search and metadata filters.

Example

document = Document(content="Refunds are paid within five working days.", meta={"topic": "refunds"})
print(document.content, document.meta)
print(len(document.id), document.id[:12])

same = Document(content="Refunds are paid within five working days.", meta={"topic": "refunds"})
print(same.id == document.id)

A Document's id is a hash of its content and metadata, so the same text with the same metadata always has the same id. That is how a store spots duplicates.

Example

from haystack import Document
from haystack.document_stores.in_memory import InMemoryDocumentStore

store = InMemoryDocumentStore()
print(store.write_documents([Document(content="Refunds are paid within five working days.")]))
print(store.count_documents())
store.write_documents([Document(content="Refunds are paid within five working days.")])

write_documents returns how many were written. Writing the same document again raises DuplicateDocumentError, the default policy, so re-running an indexing script cannot silently double your index.

Example

from haystack.document_stores.types import DuplicatePolicy

print(store.write_documents([Document(content="Refunds are paid within five working days.")], policy=DuplicatePolicy.SKIP))
print(store.write_documents([Document(content="Refunds are paid within five working days.")], policy=DuplicatePolicy.OVERWRITE))
print(store.count_documents())

SKIP ignores documents that exist, OVERWRITE replaces them. Choose one deliberately for every indexing job.

Other stores

InMemoryDocumentStore is lost when the process ends. Integrations such as elasticsearch-haystack, qdrant-haystack and pgvector-haystack provide stores with the same methods, and retrievers made for them.

Try it yourself

Change one metadata value and compare the ids.
Call store.filter_documents() with no filters.
Delete a document with store.delete_documents([document.id]).

PreviousBranches: routing with ConditionalRouter

Every expert started right here.