Why retrieval: a model does not know your documents

A model has never read this shop's refund policy. Answering from it means finding the right passage for the prompt, which is harder than matching words.

LLM Fundamentals showed a model inventing a refund policy it was never given. The fix is to give it the real one, but only the relevant part: sending every document with every question costs tokens and buries the answer. Start with the obvious way to find the relevant part, matching words:

DOCS = {
    "refunds.md": "You can get a full refund within 30 days of delivery.",
    "delivery.md": "Standard delivery takes 3 to 5 working days.",
    "lamps.md": "The LMP-204 desk lamp has a known cable fault.",
}


def keyword_search(question):
    words = {w for w in question.lower().replace("?", "").split() if len(w) > 3}
    return [name for name, text in DOCS.items() if words & set(text.lower().replace(".", "").split())]

for question in ["How long does delivery take?", "Can I get a refund?", "How do I get my money back?", "My parcel is late"]:
    print(f"{question:30} -> {keyword_search(question)}")

keyword_search keeps words longer than three letters, so how and can match nothing, and returns every file sharing a word. The first two questions find their file, and delivery also turns up refunds, whose text mentions delivery. "How do I get my money back?" finds nothing: it means refund and never says it. And "My parcel is late" finds nothing either, though it is a delivery question.

People describe the same need in different words, so search has to match meaning, not spelling. That is what embeddings do, in the next lesson. Keyword search still has a job, as lesson 13 shows with part numbers like LMP-204.

The pieces of a RAG system

• Documents: the source material, split into chunks.
• An index: the chunks stored in a form that can be searched by meaning.
• A retriever: finds the chunks that best match a question.
• A model: answers the question from those chunks, and says so when they do not contain the answer.