ID based context precision and recall

Before any judge, you can already grade the search itself. If you know which documents should have come back, comparing ids is arithmetic.

Example

import asyncio

from ragas import SingleTurnSample
from ragas.metrics import IDBasedContextPrecision, IDBasedContextRecall

sample = SingleTurnSample(
    user_input="Where is order A17?",
    retrieved_context_ids=["policy_1", "policy_9"],
    reference_context_ids=["policy_1", "policy_4"],
)
precision = asyncio.run(IDBasedContextPrecision().single_turn_ascore(sample))
recall = asyncio.run(IDBasedContextRecall().single_turn_ascore(sample))
print("precision", precision)
print("recall   ", recall)

Precision asks what share of the documents you retrieved should have been retrieved. One of the two was wanted, so 0.5.

Recall asks what share of the documents that were wanted you actually found. Again one of two, so 0.5. The two answer different questions and often move in opposite directions: retrieve more documents and recall goes up while precision goes down.

These two are worth knowing early because they are the shape of every retrieval metric in the library. The judged versions in lessons 6 and 7 ask a model the same questions about text, when you have no ids to compare.

Two APIs, and this one is the older

These metrics come from ragas.metrics and are scored with single_turn_ascore, which is asynchronous, so the snippet wraps it in asyncio.run. From the next part the course uses ragas.metrics.collections instead, which has a plain score. Lesson 16 explains why both exist.

Try it yourself

Add a third retrieved id that is not wanted, and watch precision fall while recall stays put.
Retrieve every reference id and check that both scores reach 1.

PreviousMetrics with no model: exact match and string presence

Slow is fine. Stopping is the only problem.