Evaluating retrieval

Retrieval evaluators compare what a retriever found with the documents it should have found, per question and on average. They need labels, not a model.

Example

questions = ["how long do refunds take", "can damaged items be returned", "when do parcels ship"]
expected = [
    [Document(content="Refunds are paid within five working days of receiving the return.")],
    [Document(content="Damaged items can be returned for free within 30 days.")],
    [Document(content="Parcels ship within two days and arrive by courier.")],
]

Example

retrieved = [retriever.run(query=q)["documents"] for q in questions]

recall = DocumentRecallEvaluator().run(ground_truth_documents=expected, retrieved_documents=retrieved)
mrr = DocumentMRREvaluator().run(ground_truth_documents=expected, retrieved_documents=retrieved)
print("recall", recall["score"], recall["individual_scores"])
print("mrr   ", mrr["score"], mrr["individual_scores"])

Recall: did the right document appear anywhere in the results? 1.0 for all three questions.
MRR, mean reciprocal rank: 1 divided by the position of the first right document, averaged. 1.0 means it was always first; 0.5 means second.

Evaluators match documents by content. With these three questions the retriever is perfect, which is what a first evaluation set usually shows: questions written by the person who wrote the documents use the documents' words.

Example

questions.append("I want my money back")
expected.append([Document(content="Refunds are paid within five working days of receiving the return.")])
retrieved = [retriever.run(query=q)["documents"] for q in questions]
print(DocumentRecallEvaluator().run(ground_truth_documents=expected, retrieved_documents=retrieved)["individual_scores"])

A question in a customer's words drops recall to 0 for that question. Collect real questions for the evaluation set, then use the score to compare BM25, embeddings, top_k and splitting choices.

Evaluating answers

FaithfulnessEvaluator and ContextRelevanceEvaluator ask a model whether an answer is supported by its documents, and need a real model. The DeepEval and Ragas courses cover model-judged metrics in depth.

Try it yourself

Set top_k=1 and compare MRR.
Use DocumentRecallEvaluator(mode="multi_hit") with two expected documents for one question.
Add DocumentMAPEvaluator.

PreviousAgents and tools

This is what real progress feels like.