Measuring retrieval: hit rate on labelled questions

Chunk size, top-k, keyword or hybrid: each choice is a guess until measured. Labelled questions and a hit rate turn each into a number.

questions = {
    "How do I get my money back?": "refunds.md",
    "Is the refund paid to my card?": "refunds.md",
    "My parcel still has not come": "delivery.md",
    "Can I get it tomorrow?": "delivery.md",
    "LMP-204": "lamps.md",
    "Are used bulbs refundable?": "lamps.md",
}

Six questions a customer might ask, each labelled with the file that answers it. Several are deliberately hard: no shared words, a part number, a question about bulbs that also mentions refunds.

def hit_rate(retriever):
    hits = 0
    for question, expected in questions.items():
        found = [n.metadata["file_name"] for n in retriever.retrieve(question)]
        hits += expected in found
    return hits / len(questions)

Hit rate is the share of questions where the right file appears anywhere in the retrieved results. It is one of the retrieval metrics LlamaIndex's evaluation guide describes, along with MRR, which also rewards the right result being first.

from llama_index.core.llms import MockLLM
from llama_index.core.retrievers import QueryFusionRetriever

for k in (1, 2):
    semantic = index.as_retriever(similarity_top_k=k)
    keyword = BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=k)
    hybrid = QueryFusionRetriever([semantic, keyword], llm=MockLLM(), similarity_top_k=k, num_queries=1, mode="reciprocal_rerank", use_async=False)
    print(f"top {k}:  semantic {hit_rate(semantic):.0%}  keyword {hit_rate(keyword):.0%}  hybrid {hit_rate(hybrid):.0%}")

Six numbers instead of six opinions. Keyword search misses one reworded question when it may return only one result and catches it at two; semantic and hybrid search find every file either way. With three short files the test is easy; the value of writing it is that the same function runs unchanged on a real document set, where the numbers spread apart.

Six questions is a demonstration, not a test set. As in LLM Fundamentals, a real one has dozens to hundreds of questions from real users. LlamaIndex's RetrieverEvaluator computes hit rate, MRR and more over such a set, and the Ragas and DeepEval courses score the answers as well as the retrieval.