Generative AI and Agentic AILangChain 1.4 · LangGraph 1.2 · Python 3.12 or 3.13
Dashboard
0%

Evaluating RAG

Evaluating RAG is the practice of running a pipeline on questions with known answers and scoring each run on four checks: whether the answer is correct, whether it is grounded in the retrieved documents, whether it addresses the question, and whether retrieval found useful documents.

Last updated: 08 Oct, 2026 · LangChain 1.4

A RAG app that looks right on three hand-picked questions can still fail on the fourth. An evaluation turns that guess into a table: the same questions, the same checks, run again after every change. The video builds this with LangSmith and OpenAI models; the code here keeps the dataset, the graders and the scores in plain Python so it runs with the course keys alone. The general method, with judges for any LLM app, is in Evaluating LLM apps.

The four checks

RAG evaluation: the four metrics and the plan · from the Complete Agentic AI Course video · 9:57:59 to 10:02:05
Four checks around one RAG run. Retrieval relevance compares the retrieved documents with the question. Groundedness compares the answer with the retrieved documents. Answer relevance compares the answer with the question. Correctness compares the answer with a reference answer.

The video takes the diagram from the LangSmith docs. A RAG run has four things: the question, the retrieved documents, the generated answer and, from a test set, a reference answer. Each check compares two of them:

  • Retrieval relevance compares the retrieved documents with the question: did the search bring back anything useful?
  • Groundedness compares the answer with the retrieved documents: is every claim supported by them, with nothing added from the model's memory?
  • Answer relevance compares the answer with the question: does it address what was asked?
  • Correctness compares the answer with the reference answer: does it say the same thing?

The first three need no reference answer. Correctness needs one, so the test set holds questions with ground-truth answers. The video's plan has three steps: create the RAG app, create the test data, then create the evaluators.

A RAG function that returns its documents

A rag_bot that returns the answer and the retrieved documents · from the Complete Agentic AI Course video · 10:06:27 to 10:09:50

To grade retrieval and groundedness, the function under test must return the documents as well as the answer. The video's rag_bot returns a dictionary with both, and so does the function below. It also uses a smaller model for the app, openai/gpt-oss-20b, and a larger one, openai/gpt-oss-120b, as the judge: a model that grades its own answers tends to favour them, so the grader should not be the model that wrote them. The video marks its function with LangSmith's traceable decorator and calls an OpenAI model; the version here has no tracing and calls Groq.

python
def rag_bot(question):
    docs = retriever.invoke(question)
    context = "\n\n".join(d.page_content for d in docs)
    answer = (prompt | app_model).invoke({"context": context, "question": question}).content
    return {"answer": answer, "documents": docs}

The test set

Each row holds a question, a reference answer and, for the retrieval check without a judge, a short text that must appear in the right chunk. Every reference below was taken from the papers' own pages. The last question asks for a number from a page that is not in the index, so the app cannot know it, and the rows show what each check reports when retrieval fails.

python
dataset = [
    {"question": "Which foundation model is QZhou-Embedding built upon?",
     "reference": "It is built upon Qwen2.5-7B-Instruct.", "needle": "Qwen2.5-7B-Instruct"},
    {"question": "How many parallel attention heads does the Transformer use?",
     "reference": "It uses h = 8 parallel attention layers, or heads.", "needle": "h = 8"},
    # two more rows in the full example below
]

Four graders with structured output

The correctness grader: reference answer, schema and prompt · from the Complete Agentic AI Course video · 10:13:36 to 10:18:01

The video's graders return a typed result: an explanation first and then a boolean, so the model reasons before it commits to a verdict. The same shape works with Pydantic. A grader is a function that builds one prompt from the compared items and returns the boolean. With a boolean field there is no reply text to compare, so a verdict such as Correct. cannot be misread.

python
from pydantic import BaseModel

class Grade(BaseModel):
    explanation: str      # the model explains first
    passed: bool          # then gives the verdict

def grader(instruction):
    g = judge.with_structured_output(Grade, method="json_schema")
    return lambda **kw: g.invoke(instruction + "\n\n" + "\n".join(f"{k.upper()}: {v}" for k, v in kw.items())).passed
Relevance, groundedness and retrieval relevance graders · from the Complete Agentic AI Course video · 10:20:19 to 10:24:28

The other three graders in the video differ only in the instruction and in what is compared, which is why one helper builds them all. The video builds its judge on OpenAI with strict JSON-schema output; Groq's method="json_schema" plays that part here.

python
correctness = grader("Compare the ANSWER with the REFERENCE. Pass if the answer is factually consistent with the reference; extra detail is fine.")
relevance = grader("Pass if the ANSWER addresses the QUESTION.")
groundedness = grader("Pass if every claim in the ANSWER is supported by the DOCUMENTS.")
retrieval_relevance = grader("Pass if the DOCUMENTS contain information that helps answer the QUESTION.")

Running the evaluation

The loop calls the app on each question, runs the four graders, and records the rank at which the answer text first appears among the retrieved chunks, a retrieval check that needs no model call. The setup is the four-page, two-paper store from Embeddings and vector stores, with k=3.

ExampleAPI keyStarts from the four-page store of the earlier lessons; app on gpt-oss-20b, judge on gpt-oss-120b, both on Groq
from pydantic import BaseModel
from langchain_core.prompts import ChatPromptTemplate
from langchain.chat_models import init_chat_model

retriever = store.as_retriever(search_kwargs={"k": 3})
app_model = init_chat_model("groq:openai/gpt-oss-20b", temperature=0)
judge = init_chat_model("groq:openai/gpt-oss-120b", temperature=0)
prompt = ChatPromptTemplate.from_messages([
    ("system", "Answer using only the context. If the answer is not in the context, say you do not know. One sentence."),
    ("human", "Context:\n{context}\n\nQuestion: {question}"),
])

def rag_bot(question):
    docs = retriever.invoke(question)
    context = "\n\n".join(d.page_content for d in docs)
    return {"answer": (prompt | app_model).invoke({"context": context, "question": question}).content, "documents": docs}

dataset = [
    {"question": "Which foundation model is QZhou-Embedding built upon?", "reference": "It is built upon Qwen2.5-7B-Instruct.", "needle": "Qwen2.5-7B-Instruct"},
    {"question": "How many parallel attention heads does the Transformer use?", "reference": "It uses h = 8 parallel attention layers, or heads.", "needle": "h = 8"},
    {"question": "What is the inner-layer dimensionality of the feed-forward network?", "reference": "The inner layer has dimensionality d_ff = 2048.", "needle": "2048"},
    {"question": "What BLEU score did the Transformer reach on English-to-German translation?", "reference": "28.4 BLEU on the WMT 2014 English-to-German task.", "needle": "28.4"},
]

class Grade(BaseModel):
    explanation: str
    passed: bool

def grader(instruction):
    g = judge.with_structured_output(Grade, method="json_schema")
    return lambda **kw: g.invoke(instruction + "\n\n" + "\n".join(f"{k.upper()}: {v}" for k, v in kw.items())).passed

correctness = grader("Compare the ANSWER with the REFERENCE. Pass if the answer is factually consistent with the reference; extra detail is fine.")
relevance = grader("Pass if the ANSWER addresses the QUESTION.")
groundedness = grader("Pass if every claim in the ANSWER is supported by the DOCUMENTS.")
retrieval_relevance = grader("Pass if the DOCUMENTS contain information that helps answer the QUESTION.")

rows = []
for ex in dataset:
    out = rag_bot(ex["question"])
    docs_text = "\n---\n".join(d.page_content for d in out["documents"])
    rows.append({
        "question": ex["question"][:38],
        "answer": out["answer"],
        "correct": correctness(question=ex["question"], reference=ex["reference"], answer=out["answer"]),
        "relevant": relevance(question=ex["question"], answer=out["answer"]),
        "grounded": groundedness(answer=out["answer"], documents=docs_text),
        "retrieval_ok": retrieval_relevance(question=ex["question"], documents=docs_text),
        "hit_rank": next((i for i, d in enumerate(out["documents"], 1) if ex["needle"] in d.page_content), None),
    })
for row in rows:
    print(row)
for check in ["correct", "relevant", "grounded", "retrieval_ok"]:
    print(f"{check:13} {sum(r[check] for r in rows)}/{len(rows)}")
ranks = [r["hit_rank"] for r in rows]
print("hit rate @3:", sum(r is not None for r in ranks) / len(ranks), " mean reciprocal rank:", round(sum(1 / r for r in ranks if r) / len(ranks), 3))

Reading the scores

Reading the results table: the metrics side by side · from the Complete Agentic AI Course video · 10:27:25 to 10:29:20

The video ends on the results table in LangSmith: one row for each question, one column for each evaluator, and latency and token cost beside them. The loop above prints the same kind of table from plain Python.

  • The first three rows pass every check. Each answer matches its reference, is on point, is supported by the documents, and the answer text sits in the first retrieved chunk (hit_rank 1).
  • The fourth row fails all four. The BLEU score is on a page that is not in the index, so no retrieved chunk contains it (hit_rank None, retrieval_ok False) and the app answered I do not know. That fails correctness and relevance, as expected. The judge also marked it not grounded, although the answer claims nothing; a refusal and a wrong answer deserve different labels, so judge verdicts should be read, not only counted.
  • The totals are 3 of 4 on each check. The hit rate at 3 is 0.75 and the mean reciprocal rank is 0.75: three questions found the answer at rank 1 and one found nothing.
  • Where to look. When retrieval relevance and correctness fail in the same row, search is the problem and the fix is in the index or the retriever, not in the prompt. When retrieval passes and groundedness fails, the model ignored good chunks.

Judge-based checks vs checks without a judge

Judge-based (correctness, groundedness, relevance)Label-based (hit rate, mean reciprocal rank)
NeedsA grader model call for each check and rowA text that marks the right chunk, no model
Cost per rowOne call per checkAlmost nothing
MeasuresQuality of the answer and of the retrieved textWhether search finds the labelled chunk, and how high
WeaknessThe judge can be wrong or lenientOnly as good as the labels

Where you use RAG evaluation

  • Before a change ships. Re-run the same rows after changing chunk size, k or the prompt, and compare the tables.
  • Finding which half failed. A pass on retrieval relevance and a fail on groundedness means the model ignored good chunks; the reverse means search is the problem.
  • Regression tests. A fixed question set that runs in a build and fails when the pass counts drop.
Watch out. Four questions show how the checks work, not how good the app is. A real test set needs dozens of questions drawn from real users, and a judge's verdicts should be spot-checked by reading a sample. A judge from the same model family as the app tends to favour its own style, which is why the app and the judge differ above. Libraries such as RAGAS and DeepEval package these metrics with larger test sets.
Try it yourself
  • Change k=3 to k=1 and run again: which checks change, and does the hit rate move?
  • Add a row whose answer is in the papers but worded in a very different way from the question.
  • Swap the judge to openai/gpt-oss-20b and compare the verdicts with the larger judge row by row.

This is what real progress feels like.