Updating documents: refresh and delete

Policies change. An index needs to update edited documents and remove deleted ones without a full rebuild, or it keeps quoting old rules.

Example

from llama_index.core import Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

Settings.embed_model = HuggingFaceEmbedding(model_name="sentence-transformers/all-MiniLM-L6-v2")
import os

from llama_index.core import SimpleDirectoryReader, VectorStoreIndex


def only_name(path):
    return {"file_name": os.path.basename(path)}


def load():
    return SimpleDirectoryReader("help", file_metadata=only_name, filename_as_id=True).load_data()


index = VectorStoreIndex.from_documents(load())

filename_as_id=True again gives each document a stable id from its file name, which is what lets the index recognise an edited file as the same document.

Example

with open("help/refunds.md", "w") as f:
    f.write("# Refunds\n\nYou can get a full refund within 60 days of delivery.\n")

changed = index.refresh_ref_docs(load())
print("changed:", changed)
retriever = index.as_retriever(similarity_top_k=1)
print(retriever.retrieve("How long do I have to ask for a refund?")[0].get_content().replace("\n", " "))

refresh_ref_docs compares each document with what the index holds, re-processes the ones whose content changed, and returns a list of which did. Only the refund file changed, and retrieval now returns the 60-day policy.

A document that was removed

Example

lamp_id = next(doc.doc_id for doc in load() if doc.metadata["file_name"] == "lamps.md")
index.delete_ref_doc(lamp_id, delete_from_docstore=True)
print([n.metadata["file_name"] for n in index.as_retriever(similarity_top_k=3).retrieve("LMP-204 cable fault")])

delete_ref_doc removes every chunk that came from that document. The lamp question now retrieves only unrelated files, which, with a cutoff from lesson 11, becomes a refusal instead of an outdated answer.

Refreshing on a schedule, and deleting documents the source system no longer has, is part of running a RAG system. So is the deletion duty in the customer data and PII topic: a document removed for legal reasons must leave the index too.

Try it yourself

Add a new file to help and call refresh_ref_docs. What does it return for the new file?
Call refresh_ref_docs twice without changes.
Delete a document and persist the index, then load it and check the chunks are gone.

PreviousPersisting an index: not embedding twice

Slow is fine. Stopping is the only problem.