ID-based context precision and recall

ID-based context precision and recall are RAGAS metrics that grade a retriever by comparing the ids of the chunks it returned with the ids of the chunks it should have returned, with no judge model.

Last updated: 29 Sep, 2026 · RAGAS 0.4.3

Before any judge, you can already grade the search. If you know which catalog entries a question needs, comparing ids is counting. These two metrics are also the shape of every retrieval metric that comes later.

The IDBasedContextPrecision and IDBasedContextRecall API

python

from ragas import SingleTurnSample
from ragas.metrics import IDBasedContextPrecision, IDBasedContextRecall

sample = SingleTurnSample(retrieved_context_ids=[...], reference_context_ids=[...])
await IDBasedContextPrecision().single_turn_ascore(sample)  # share of retrieved ids that were wanted
await IDBasedContextRecall().single_turn_ascore(sample)     # share of wanted ids that were retrieved

These two live in ragas.metrics, the older metrics API, and are scored with single_turn_ascore, which is asynchronous. A script wraps it in asyncio.run. Collections and legacy metrics explains the two APIs.

The ids g004 needs

g004 asks about shipping and returns, so the right chunks are the shipping policy and the return policy.

python

reference_ids = ["policy_002", "policy_001"]  # shipping policy, return policy

The ids the app retrieved

The app returns chunk texts. Looking each text up in the catalog gives its id.

python

contexts = retrieve(question, top_k=3)
retrieved_ids = [item["id"] for text in contexts for item in CATALOG if item["content"] == text]

Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports these files. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep them in the same folder.

written in TechNest RAG app
written in TechNest RAG app

View the code here

technest.py

import json
import os

import numpy as np
from google import genai
from openai import OpenAI

gemini = genai.Client()  # reads GOOGLE_API_KEY
groq = OpenAI(api_key=os.environ["GROQ_API_KEY"], base_url="https://api.groq.com/openai/v1")
EMBED_MODEL = "gemini-embedding-2"
CHAT_MODEL = "qwen/qwen3.8-27b"

SYSTEM_PROMPT = """You are a helpful customer support assistant for TechNest, an online electronics store.
Answer the customer's question using ONLY the information provided in the context below.
If the context does not contain enough information to answer fully, say so honestly.
Keep your answer concise, factual, and friendly. Do not invent any details not present in the context.
Reply in two or three plain sentences, with no lists or tables."""

with open("catalog.json", encoding="utf-8") as f:
    CATALOG = json.load(f)


def embed(texts):
    # gemini-embedding-2 turns everything in one call into one embedding, so send one text per call
    vectors = np.array([gemini.models.embed_content(model=EMBED_MODEL, contents=t).embeddings[0].values for t in texts])
    return vectors / np.linalg.norm(vectors, axis=1, keepdims=True)


DOC_VECTORS = embed([f"{item['title']}. {item['content']}" for item in CATALOG])


def retrieve(question, top_k=3):
    scores = DOC_VECTORS @ embed([question])[0]
    best = np.argsort(scores)[::-1][:top_k]
    return [CATALOG[i]["content"] for i in best]


def generate(question, contexts):
    context_block = "\n\n".join(f"[{i+1}] {c}" for i, c in enumerate(contexts))
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": f"Context:\n{context_block}\n\nCustomer question: {question}"},
    ]
    response = groq.chat.completions.create(model=CHAT_MODEL, messages=messages, temperature=0)
    return response.choices[0].message.content.strip()


def answer(question, top_k=3):
    contexts = retrieve(question, top_k)
    return generate(question, contexts), contexts

catalog.json

[
  {
    "id": "prod_001",
    "category": "product",
    "title": "ProBook X1 Laptop",
    "content": "The TechNest ProBook X1 is a 14-inch laptop featuring an Intel Core i7-13th Gen processor, 16GB DDR5 RAM, and a 512GB NVMe SSD. It has a battery life of 12 hours, weighs 1.4kg, and comes with a backlit keyboard. Price: $1,299. Includes a 2-year manufacturer warranty."
  },
  {
    "id": "prod_002",
    "category": "product",
    "title": "PixelPhone 15",
    "content": "The TechNest PixelPhone 15 is a 6.7-inch AMOLED smartphone with a 50MP triple camera system, 8GB RAM, 256GB storage, and a 5,000mAh battery supporting 65W fast charging. Available in Midnight Black and Arctic White. Price: $899. Includes a 1-year warranty."
  },
  {
    "id": "prod_003",
    "category": "product",
    "title": "SoundPods Pro",
    "content": "The TechNest SoundPods Pro are true wireless earbuds with active noise cancellation (ANC), 8 hours of playback per charge plus 24 hours with the case, and IPX4 water resistance. They connect via Bluetooth 5.3 and support multipoint pairing with two devices simultaneously. Price: $149."
  },
  {
    "id": "prod_004",
    "category": "product",
    "title": "UltraTab S2 Tablet",
    "content": "The TechNest UltraTab S2 is a 11-inch tablet powered by a Snapdragon 870 processor with 8GB RAM and 128GB storage expandable via microSD. It features a 120Hz display, a 7,500mAh battery, and supports the TechNest Stylus Pen sold separately. Price: $549. Includes a 1-year warranty."
  },
  {
    "id": "prod_005",
    "category": "product",
    "title": "SmartWatch X",
    "content": "The TechNest SmartWatch X features continuous heart rate monitoring, SpO2 tracking, GPS, and 7-day battery life. It is water-resistant up to 50 metres. Compatible with both Android and iOS. Price: $299. Includes a 1-year warranty and a free extra silicone band."
  },
  {
    "id": "prod_006",
    "category": "product",
    "title": "ProCam 4K Action Camera",
    "content": "The TechNest ProCam 4K shoots 4K video at 60fps and 20MP photos. It is waterproof up to 10 metres without a case, has built-in image stabilisation (EIS), and includes a touch screen. Battery life is 90 minutes of 4K recording. Price: $229. Includes a 1-year warranty."
  },
  {
    "id": "prod_007",
    "category": "product",
    "title": "BassBuds Max Headphones",
    "content": "The TechNest BassBuds Max are over-ear wireless headphones with 40-hour battery life, hybrid active noise cancellation, and a premium 40mm driver for deep bass. They fold flat for travel and include a carrying case. Price: $199. Compatible with all Bluetooth devices."
  },
  {
    "id": "prod_008",
    "category": "product",
    "title": "SoundBar 360",
    "content": "The TechNest SoundBar 360 is a 2.1 soundbar with a 120W output, built-in subwoofer, Dolby Atmos support, and HDMI ARC connectivity. It also supports Bluetooth streaming and has an optical audio input. Dimensions: 90cm wide. Price: $349. Includes a 2-year warranty."
  },
  {
    "id": "policy_001",
    "category": "policy",
    "title": "Return Policy",
    "content": "TechNest accepts returns within 30 days of the original purchase date. Items must be in their original packaging with all accessories included. Customers are responsible for return shipping costs unless the item arrives defective or damaged. Refunds are processed within 5 to 7 business days of receiving the returned item. Digital downloads and opened software are non-refundable."
  },
  {
    "id": "policy_002",
    "category": "policy",
    "title": "Shipping Policy",
    "content": "TechNest offers free standard shipping on all orders over $50 within the continental US. Standard shipping takes 3 to 5 business days. Expedited shipping (1 to 2 business days) is available for $9.99. Same-day delivery is available in select cities for $19.99. Orders placed before 2pm local time are dispatched the same day."
  },
  {
    "id": "policy_003",
    "category": "policy",
    "title": "Warranty Policy",
    "content": "All TechNest products include a minimum 1-year manufacturer warranty covering defects in materials and workmanship. The ProBook X1 and SoundBar 360 include a 2-year warranty. Warranty does not cover physical damage, water damage (unless the product is rated waterproof), or damage from unauthorised modifications. To make a warranty claim, contact support@technest.com with your order number and a description of the issue."
  },
  {
    "id": "policy_004",
    "category": "policy",
    "title": "Payment Policy",
    "content": "TechNest accepts Visa, Mastercard, American Express, PayPal, and Apple Pay. All transactions are encrypted using 256-bit SSL. Buy Now Pay Later is available via Klarna for orders over $100, with 0% interest for 3 monthly instalments. TechNest does not store full card details — payments are processed securely by Stripe."
  },
  {
    "id": "faq_001",
    "category": "faq",
    "title": "Order Tracking",
    "content": "To track your order, visit technest.com/orders and enter your order number and email address. A shipping confirmation email with a tracking link is sent within 24 hours of dispatch. If you have not received your tracking email after 48 hours, check your spam folder or contact support@technest.com."
  },
  {
    "id": "faq_002",
    "category": "faq",
    "title": "International Shipping",
    "content": "TechNest ships to over 40 countries. International shipping rates start at $14.99 and delivery takes 7 to 14 business days. Import duties and taxes are the responsibility of the customer and are not included in the product price. Free shipping promotions apply to US orders only."
  },
  {
    "id": "faq_003",
    "category": "faq",
    "title": "Bulk and Business Orders",
    "content": "TechNest offers volume discounts for businesses purchasing 10 or more units of any single product. Discounts range from 10% for 10 to 49 units up to 25% for 100 or more units. Contact business@technest.com with your requirements for a custom quote. A dedicated account manager is assigned for orders over $10,000."
  }
]

Grading the TechNest retriever on g004

The same question at top_k=3 and at top_k=1. The search is real; only the counting is plain Python.

ExampleAPI key

import asyncio

from ragas import SingleTurnSample
from ragas.metrics import IDBasedContextPrecision, IDBasedContextRecall
from technest import CATALOG, retrieve

question = "What are TechNest's shipping options and how long do returns take to process?"
reference_ids = ["policy_002", "policy_001"]

for top_k in [3, 1]:
    contexts = retrieve(question, top_k=top_k)
    retrieved_ids = [item["id"] for text in contexts for item in CATALOG if item["content"] == text]
    sample = SingleTurnSample(retrieved_context_ids=retrieved_ids, reference_context_ids=reference_ids)
    precision = asyncio.run(IDBasedContextPrecision().single_turn_ascore(sample))
    recall = asyncio.run(IDBasedContextRecall().single_turn_ascore(sample))
    print(f"top_k={top_k}  retrieved={retrieved_ids}")
    print(f"          precision={precision:.2f}  recall={recall:.2f}")

Output

top_k=3  retrieved=['policy_002', 'policy_001', 'faq_002']
          precision=0.67  recall=1.00
top_k=1  retrieved=['policy_002']
          precision=1.00  recall=0.50

How the two numbers moved

Precision at top_k=3 is 0.67. The retriever returned the shipping policy, the return policy and the international shipping FAQ; two of the three were wanted.
Recall at top_k=3 is 1.00. Both wanted policies are in the list.
Lowering top_k to 1 keeps only the shipping policy. Precision rises to 1.00, because nothing unwanted came back, and recall falls to 0.50, because the return policy is gone.

Precision vs recall

	Precision	Recall
Asks	How much of what I fetched was useful?	How much of what was useful did I fetch?
Hurt by	Extra, unrelated chunks	A missing chunk
Usually moves when top_k grows	Down	Up

When id-based metrics are enough

When every golden lists the ids of the chunks it needs, so no judge is needed to decide relevance.
When you tune top_k or swap embedding models and want a quick, free score for the search alone.

Watch out. Ids only work if they are stable. If you re-chunk the documents and the ids change, every reference_context_ids list in your goldens is silently wrong and the scores drop for no real reason.

Previous: Exact match and string presence
Next: LLM as a judge
Reference: Context precision (ID based)

Try it yourself

Run the loop with top_k=5 as well and watch precision fall while recall stays at 1.
Grade g002, the ProBook question, whose only wanted id is prod_001.

PreviousExact match and string presence

You understood something today that you didn't yesterday.

ID-based context precision and recall

The IDBasedContextPrecision and IDBasedContextRecall API

The ids g004 needs

The ids the app retrieved

Grading the TechNest retriever on g004

How the two numbers moved

Precision vs recall

When id-based metrics are enough

Related