Evaluating RAG
Evaluating RAG is the practice of running a pipeline on questions with known answers and scoring each run on four checks: whether the answer is correct, whether it is grounded in the retrieved documents, whether it addresses the question, and whether retrieval found useful documents.
Last updated: 08 Oct, 2026 · LangChain 1.4
A RAG app that looks right on three hand-picked questions can still fail on the fourth. An evaluation turns that guess into a table: the same questions, the same checks, run again after every change. The video builds this with LangSmith and OpenAI models; the code here keeps the dataset, the graders and the scores in plain Python so it runs with the course keys alone. The general method, with judges for any LLM app, is in Evaluating LLM apps.
The four checks
The video takes the diagram from the LangSmith docs. A RAG run has four things: the question, the retrieved documents, the generated answer and, from a test set, a reference answer. Each check compares two of them:
- Retrieval relevance compares the retrieved documents with the question: did the search bring back anything useful?
- Groundedness compares the answer with the retrieved documents: is every claim supported by them, with nothing added from the model's memory?
- Answer relevance compares the answer with the question: does it address what was asked?
- Correctness compares the answer with the reference answer: does it say the same thing?
The first three need no reference answer. Correctness needs one, so the test set holds questions with ground-truth answers. The video's plan has three steps: create the RAG app, create the test data, then create the evaluators.
A RAG function that returns its documents
To grade retrieval and groundedness, the function under test must return the documents as well as the answer. The video's rag_bot returns a dictionary with both, and so does the function below. It also uses a smaller model for the app, openai/gpt-oss-20b, and a larger one, openai/gpt-oss-120b, as the judge: a model that grades its own answers tends to favour them, so the grader should not be the model that wrote them. The video marks its function with LangSmith's traceable decorator and calls an OpenAI model; the version here has no tracing and calls Groq.
def rag_bot(question):
docs = retriever.invoke(question)
context = "\n\n".join(d.page_content for d in docs)
answer = (prompt | app_model).invoke({"context": context, "question": question}).content
return {"answer": answer, "documents": docs}The test set
Each row holds a question, a reference answer and, for the retrieval check without a judge, a short text that must appear in the right chunk. Every reference below was taken from the papers' own pages. The last question asks for a number from a page that is not in the index, so the app cannot know it, and the rows show what each check reports when retrieval fails.
dataset = [
{"question": "Which foundation model is QZhou-Embedding built upon?",
"reference": "It is built upon Qwen2.5-7B-Instruct.", "needle": "Qwen2.5-7B-Instruct"},
{"question": "How many parallel attention heads does the Transformer use?",
"reference": "It uses h = 8 parallel attention layers, or heads.", "needle": "h = 8"},
# two more rows in the full example below
]Four graders with structured output
The video's graders return a typed result: an explanation first and then a boolean, so the model reasons before it commits to a verdict. The same shape works with Pydantic. A grader is a function that builds one prompt from the compared items and returns the boolean. With a boolean field there is no reply text to compare, so a verdict such as Correct. cannot be misread.
from pydantic import BaseModel
class Grade(BaseModel):
explanation: str # the model explains first
passed: bool # then gives the verdict
def grader(instruction):
g = judge.with_structured_output(Grade, method="json_schema")
return lambda **kw: g.invoke(instruction + "\n\n" + "\n".join(f"{k.upper()}: {v}" for k, v in kw.items())).passedThe other three graders in the video differ only in the instruction and in what is compared, which is why one helper builds them all. The video builds its judge on OpenAI with strict JSON-schema output; Groq's method="json_schema" plays that part here.
correctness = grader("Compare the ANSWER with the REFERENCE. Pass if the answer is factually consistent with the reference; extra detail is fine.")
relevance = grader("Pass if the ANSWER addresses the QUESTION.")
groundedness = grader("Pass if every claim in the ANSWER is supported by the DOCUMENTS.")
retrieval_relevance = grader("Pass if the DOCUMENTS contain information that helps answer the QUESTION.")Running the evaluation
The loop calls the app on each question, runs the four graders, and records the rank at which the answer text first appears among the retrieved chunks, a retrieval check that needs no model call. The setup is the four-page, two-paper store from Embeddings and vector stores, with k=3.
from pydantic import BaseModel
from langchain_core.prompts import ChatPromptTemplate
from langchain.chat_models import init_chat_model
retriever = store.as_retriever(search_kwargs={"k": 3})
app_model = init_chat_model("groq:openai/gpt-oss-20b", temperature=0)
judge = init_chat_model("groq:openai/gpt-oss-120b", temperature=0)
prompt = ChatPromptTemplate.from_messages([
("system", "Answer using only the context. If the answer is not in the context, say you do not know. One sentence."),
("human", "Context:\n{context}\n\nQuestion: {question}"),
])
def rag_bot(question):
docs = retriever.invoke(question)
context = "\n\n".join(d.page_content for d in docs)
return {"answer": (prompt | app_model).invoke({"context": context, "question": question}).content, "documents": docs}
dataset = [
{"question": "Which foundation model is QZhou-Embedding built upon?", "reference": "It is built upon Qwen2.5-7B-Instruct.", "needle": "Qwen2.5-7B-Instruct"},
{"question": "How many parallel attention heads does the Transformer use?", "reference": "It uses h = 8 parallel attention layers, or heads.", "needle": "h = 8"},
{"question": "What is the inner-layer dimensionality of the feed-forward network?", "reference": "The inner layer has dimensionality d_ff = 2048.", "needle": "2048"},
{"question": "What BLEU score did the Transformer reach on English-to-German translation?", "reference": "28.4 BLEU on the WMT 2014 English-to-German task.", "needle": "28.4"},
]
class Grade(BaseModel):
explanation: str
passed: bool
def grader(instruction):
g = judge.with_structured_output(Grade, method="json_schema")
return lambda **kw: g.invoke(instruction + "\n\n" + "\n".join(f"{k.upper()}: {v}" for k, v in kw.items())).passed
correctness = grader("Compare the ANSWER with the REFERENCE. Pass if the answer is factually consistent with the reference; extra detail is fine.")
relevance = grader("Pass if the ANSWER addresses the QUESTION.")
groundedness = grader("Pass if every claim in the ANSWER is supported by the DOCUMENTS.")
retrieval_relevance = grader("Pass if the DOCUMENTS contain information that helps answer the QUESTION.")
rows = []
for ex in dataset:
out = rag_bot(ex["question"])
docs_text = "\n---\n".join(d.page_content for d in out["documents"])
rows.append({
"question": ex["question"][:38],
"answer": out["answer"],
"correct": correctness(question=ex["question"], reference=ex["reference"], answer=out["answer"]),
"relevant": relevance(question=ex["question"], answer=out["answer"]),
"grounded": groundedness(answer=out["answer"], documents=docs_text),
"retrieval_ok": retrieval_relevance(question=ex["question"], documents=docs_text),
"hit_rank": next((i for i, d in enumerate(out["documents"], 1) if ex["needle"] in d.page_content), None),
})
for row in rows:
print(row)
for check in ["correct", "relevant", "grounded", "retrieval_ok"]:
print(f"{check:13} {sum(r[check] for r in rows)}/{len(rows)}")
ranks = [r["hit_rank"] for r in rows]
print("hit rate @3:", sum(r is not None for r in ranks) / len(ranks), " mean reciprocal rank:", round(sum(1 / r for r in ranks if r) / len(ranks), 3)){'question': 'Which foundation model is QZhou-Embedd', 'answer': 'QZhou‑Embedding is built upon the Qwen2.5‑7B‑Instruct foundation model.', 'correct': True, 'relevant': True, 'grounded': True, 'retrieval_ok': True, 'hit_rank': 1}
{'question': 'How many parallel attention heads does', 'answer': 'The Transformer uses 8 parallel attention heads.', 'correct': True, 'relevant': True, 'grounded': True, 'retrieval_ok': True, 'hit_rank': 1}
{'question': 'What is the inner-layer dimensionality', 'answer': 'The inner‑layer dimensionality of the feed‑forward network is 2048.', 'correct': True, 'relevant': True, 'grounded': True, 'retrieval_ok': True, 'hit_rank': 1}
{'question': 'What BLEU score did the Transformer re', 'answer': 'I do not know.', 'correct': False, 'relevant': False, 'grounded': False, 'retrieval_ok': False, 'hit_rank': None}
correct 3/4
relevant 3/4
grounded 3/4
retrieval_ok 3/4
hit rate @3: 0.75 mean reciprocal rank: 0.75Reading the scores
The video ends on the results table in LangSmith: one row for each question, one column for each evaluator, and latency and token cost beside them. The loop above prints the same kind of table from plain Python.
- The first three rows pass every check. Each answer matches its reference, is on point, is supported by the documents, and the answer text sits in the first retrieved chunk (
hit_rank1). - The fourth row fails all four. The BLEU score is on a page that is not in the index, so no retrieved chunk contains it (
hit_rankNone,retrieval_okFalse) and the app answeredI do not know.That fails correctness and relevance, as expected. The judge also marked it not grounded, although the answer claims nothing; a refusal and a wrong answer deserve different labels, so judge verdicts should be read, not only counted. - The totals are 3 of 4 on each check. The hit rate at 3 is 0.75 and the mean reciprocal rank is 0.75: three questions found the answer at rank 1 and one found nothing.
- Where to look. When retrieval relevance and correctness fail in the same row, search is the problem and the fix is in the index or the retriever, not in the prompt. When retrieval passes and groundedness fails, the model ignored good chunks.
Judge-based checks vs checks without a judge
| Judge-based (correctness, groundedness, relevance) | Label-based (hit rate, mean reciprocal rank) | |
|---|---|---|
| Needs | A grader model call for each check and row | A text that marks the right chunk, no model |
| Cost per row | One call per check | Almost nothing |
| Measures | Quality of the answer and of the retrieved text | Whether search finds the labelled chunk, and how high |
| Weakness | The judge can be wrong or lenient | Only as good as the labels |
Where you use RAG evaluation
- Before a change ships. Re-run the same rows after changing chunk size,
kor the prompt, and compare the tables. - Finding which half failed. A pass on retrieval relevance and a fail on groundedness means the model ignored good chunks; the reverse means search is the problem.
- Regression tests. A fixed question set that runs in a build and fails when the pass counts drop.
Related
- Previous: Vectorless RAG
- Next: Project: chat with your documents
- See also: Evaluating LLM apps
- Go deeper: RAGAS and DeepEval
- Change
k=3tok=1and run again: which checks change, and does the hit rate move? - Add a row whose answer is in the papers but worded in a very different way from the question.
- Swap the judge to
openai/gpt-oss-20band compare the verdicts with the larger judge row by row.
This is what real progress feels like.