Retriever metrics: precision, recall and relevancy

The last two lessons graded the answer. When an answer is bad because the bot was handed the wrong documents, fixing the prompt will not help. Three metrics grade the search instead.

from deepeval.metrics import (ContextualPrecisionMetric, ContextualRecallMetric,
                              ContextualRelevancyMetric)

case = LLMTestCase(
    input=question,
    actual_output="Order A17 shipped on 3 March.",
    expected_output="Order A17 shipped on 3 March by courier.",
    retrieval_context=["Refunds are paid within 5 working days.",
                       "Order A17 shipped on 3 March by courier."],
)

The same question, with the useful document second and an irrelevant one first. The expected answer is there too, because two of these three metrics need to know what a complete answer contains.

for metric in [ContextualPrecisionMetric(model=PretendJudge()),
               ContextualRecallMetric(model=PretendJudge()),
               ContextualRelevancyMetric(model=PretendJudge())]:
    metric.measure(case)
    print(metric.__name__, metric.score)

Contextual precision asks whether the useful documents came before the useless ones. It needs the expected answer to know which is which. A relevant document ranked second costs score, because models pay most attention to what they see first.

Contextual recall asks whether what was retrieved contains everything the expected answer says. It is the metric that catches a search that missed a document altogether.

Contextual relevancy asks how much of what came back was about the question at all. Half of this context is about refunds, so half of it was noise.

A better search

case.retrieval_context = ["Order A17 shipped on 3 March by courier."]
for metric in [ContextualPrecisionMetric(model=PretendJudge()),
               ContextualRecallMetric(model=PretendJudge()),
               ContextualRelevancyMetric(model=PretendJudge())]:
    metric.measure(case)
    print(metric.__name__, metric.score)

One document, the right one, first. All three go to 1, and nothing about the bot's answer changed.

What the judge was asked

@answers("contextual_precision.Verdicts")
def precision_verdicts(prompt, judge):
    expected = section(prompt, "Expected output")
    return {"verdicts": [
        {"verdict": "yes" if words(doc) & words(expected) else "no", "reason": doc}
        for doc in listed(prompt, "Retrieval Context")]}

@answers("contextual_recall.Verdicts")
def recall_verdicts(prompt, judge):
    documents = " ".join(listed(prompt, "Retrieval Context"))
    return {"verdicts": [
        {"verdict": "yes" if share(s, documents) == 1 else "no", "reason": s}
        for s in sentences(section(prompt, "Expected Output"))]}

@answers("contextual_relevancy.ContextualRelevancyVerdicts")
def relevancy_of_context(prompt, judge):
    question = section(prompt, "Input")
    return {"verdicts": [
        {"statement": s, "verdict": "yes" if words(s) & words(question) else "no"}
        for s in sentences(section(prompt, "Context"))]}

Precision judges each document against the expected answer, recall judges each sentence of the expected answer against the documents, and relevancy judges each sentence of the documents against the question. Same three pieces, compared in three directions.

Which one to reach for

Those five are the standard RAG set, and the reason DeepEval splits them up: a single score cannot tell you whether to fix the search or the prompt.