Context precision: was the useful document first

Recall asks whether the right document came back. Precision asks where in the list it came back, because a model reads the top of its context far more carefully than the bottom.

from pretend_ragas import PretendJudge
from ragas.metrics.collections import ContextPrecision

precision = ContextPrecision(llm=PretendJudge())
useful = "Order A17 shipped on 3 March by courier."
noise = "Refunds are paid within 5 working days."

for context in [[useful], [useful, noise], [noise, useful]]:
    result = precision.score(user_input="Where is order A17?", reference=useful, retrieved_contexts=context)
    print(round(result.value, 4), "|", context)

One useful document alone scores 1. The useful document first with noise after it still scores 1. Put the noise first and the score halves, for exactly the same set of documents.

That is the metric's whole idea: it averages precision at each position, so a relevant document ranked second counts for less than one ranked first. It is the score that tells you to add a reranker rather than to change the prompt.

What the judge was asked

@answers("ContextPrecisionOutput")
def is_this_document_useful(data, model):
    useful = bool(words(data.get("context", "")) & words(data.get("answer", "")))
    return {"verdict": int(useful),
            "reason": "shares words with the answer" if useful else "unrelated to the answer"}

One question per document: is this one useful for the answer? RAGAS does the ranking arithmetic on top of those yes or no answers, which is why the stand-in never sees a position.