Context recall: did the search find what was needed

Faithfulness graded the answer. Context recall grades the search, by asking whether what came back contains everything the right answer needs.

from pretend_ragas import PretendJudge
from ragas.metrics.collections import ContextRecall

recall = ContextRecall(llm=PretendJudge())
reference = "Order A17 shipped on 3 March by courier."

for context in [["Order A17 shipped on 3 March by courier."],
                ["Refunds are paid within 5 working days."],
                ["Order A17 shipped on 3 March by courier.", "Refunds are paid within 5 working days."]]:
    result = recall.score(user_input="Where is order A17?", retrieved_contexts=context, reference=reference)
    print(result.value, "|", context)

With the right document, recall is 1. With only the refunds policy, nothing in the reference is covered and recall is 0. With both documents it is 1 again: recall does not care that half of what you retrieved was useless.

The judge is asked to break the reference into claims, and then to say which of them the context supports. So recall needs a reference, which means it is a metric for development, not for live traffic where nobody has written the right answer down.

What the judge was asked

@answers("ContextRecallOutput")
def check_the_reference(data, model):
    context = data.get("context", "")
    return {"classifications": [attributed_for(s, context)
                                for s in sentences(data.get("answer", ""))]}

The same shape as faithfulness, pointed the other way: the claims come from the reference, not from the answer, and each is looked for in what was retrieved.

def attributed_for(statement, context):
    """One claim from the right answer, looked for in what was retrieved."""
    found = supported(statement, context)
    return {"statement": statement, "attributed": int(found),
            "reason": "found in the context" if found else "missing from the context"}