Generative AI and Agentic AILangChain 1.4 · LangGraph 1.2 · Python 3.12 or 3.13
Dashboard
0%

Evaluating LLM apps

LLM evaluation is the practice of running an LLM app on a fixed set of test cases and scoring its answers, so that a change of model, prompt or retriever is judged by numbers instead of by feel.

Last updated: 08 Oct, 2026 · LangChain 1.4

An LLM app answers differently each time it is asked, and a prompt change that fixes one question can break three others. An evaluation turns "it seems better" into a table that can be compared from one run to the next. This lesson builds one as a plain Python loop with a judge model, which needs no account beyond the keys already set up.

Why evaluate a chatbot, and the four steps · from the Complete Agentic AI Course video · 9:25:59 to 9:30:31

Choosing between models with data

The video starts from a simple chatbot and the questions that come up once it exists. Which model should it use? Cost is one factor, but accuracy on the specific use case matters more. How do you decide that an output is good enough? That needs some ground truth: for a given input, the output you expect. And who does the comparing? The video's answer is LLM as a judge: a second model, given a prompt, that compares each response with the ground truth and gives a verdict.

The video lays the work out in four steps:

  1. Gather data points. Each one is an input and the output expected for it.
  2. Build the judge. A prompt that tells an LLM how to grade a response against the expected output.
  3. Define the metrics. What each score means: for example, correct or not, concise or not.
  4. Compare models. Run the same data points through each candidate and keep the one with the best metrics.
Data points holding a question and a reference answer go through the app under test, once for model A and once for model B. Each response enters a scoring box with a judge model that returns true or false for correctness and a rule in code that checks length, and the scores for each model come out the other side. A dashed arrow carries the reference answers straight to the scoring box.

Writing the data points

A data point is a question with a short reference answer. These are the five from the video: five questions about well-known tools and companies, each answered in one phrase.

python
dataset = [
    {"question": "What is LangChain?", "answer": "A framework for building LLM applications"},
    {"question": "What is LangSmith?", "answer": "A platform for observing and evaluating LLM applications"},
    {"question": "What is OpenAI?", "answer": "A company that creates Large Language Models"},
    {"question": "What is Google?", "answer": "A technology company known for search"},
    {"question": "What is Mistral?", "answer": "A company that creates Large Language Models"},
]

Writing a judge and a metric

The grade schema and the judge model

The judge returns structured output (the Structured output lesson covers it). The schema has two fields in a deliberate order: explanation first, then correct. The model writes its reasoning before it commits to a verdict, and the verdict arrives as a real True or False, so the code compares nothing against a string. Temperature 0 keeps the grading as steady as a model allows.

python
from typing import Annotated
from typing_extensions import TypedDict
from langchain.chat_models import init_chat_model

class Grade(TypedDict):
    explanation: Annotated[str, ..., "Explain your reasoning for the grade"]
    correct: Annotated[bool, ..., "True if the predicted answer agrees with the real answer"]

judge = init_chat_model("groq:openai/gpt-oss-120b", temperature=0).with_structured_output(Grade)

The correctness function

The function builds the grading prompt from the question, the reference answer and the response, then asks the judge. The system message is the video's: an expert professor grading students' answers.

python
def correctness(question, reference, response):
    prompt = (f"You are grading the following question:\n{question}\n"
              f"Here is the real answer:\n{reference}\n"
              f"You are grading the following predicted answer:\n{response}")
    return judge.invoke([
        ("system", "You are an expert professor specialized in grading students' answers to questions."),
        ("user", prompt),
    ])                                   # a dict: {"explanation": ..., "correct": True or False}

A rule-based metric

Not every metric needs a model. The video's second metric, concision, is a length check: the response must be shorter than twice the reference answer. It costs nothing and returns the same result every time, but it measures length and not quality.

python
def concision(reference, response):
    return len(response) < 2 * len(reference)      # shorter than twice the reference answer
Running the evaluation and reading the experiment · from the Complete Agentic AI Course video · 9:50:05 to 9:54:25

The video runs this with LangSmith: its client.evaluate calls the app for every data point in a dataset stored on LangSmith, applies both metrics and shows each experiment on a results page. Its two models are OpenAI's gpt-4o-mini and gpt-4-turbo. The loop below does the same four steps in plain Python on Groq's models, and prints the table in the terminal, so no account is needed.

Running two models through the same data points

The two models under test are gpt-oss-20b and qwen3.8-27b, both served by Groq. The judge is a third model, gpt-oss-120b, so no model grades its own answers. Every response is graded on correctness by the judge and on concision by the rule. For a wrong verdict the loop also prints the reply and the judge's explanation, because the reason behind a failure is what tells you what to fix.

ExampleAPI keyLive run on Groq, with gpt-oss-120b as the judge
from typing import Annotated
from typing_extensions import TypedDict
from langchain.chat_models import init_chat_model

dataset = [
    {"question": "What is LangChain?", "answer": "A framework for building LLM applications"},
    {"question": "What is LangSmith?", "answer": "A platform for observing and evaluating LLM applications"},
    {"question": "What is OpenAI?", "answer": "A company that creates Large Language Models"},
    {"question": "What is Google?", "answer": "A technology company known for search"},
    {"question": "What is Mistral?", "answer": "A company that creates Large Language Models"},
]

class Grade(TypedDict):
    explanation: Annotated[str, ..., "Explain your reasoning for the grade"]
    correct: Annotated[bool, ..., "True if the predicted answer agrees with the real answer"]

judge = init_chat_model("groq:openai/gpt-oss-120b", temperature=0).with_structured_output(Grade)

def correctness(question, reference, response):
    prompt = (f"You are grading the following question:\n{question}\n"
              f"Here is the real answer:\n{reference}\n"
              f"You are grading the following predicted answer:\n{response}")
    return judge.invoke([
        ("system", "You are an expert professor specialized in grading students' answers to questions."),
        ("user", prompt),
    ])

def concision(reference, response):
    return len(response) < 2 * len(reference)

instructions = "Respond to the user's question in a short, concise manner (one short sentence)."
for name in ["groq:openai/gpt-oss-20b", "groq:qwen/qwen3.8-27b"]:
    app = init_chat_model(name, temperature=0)
    right = short = 0
    for ex in dataset:
        reply = app.invoke([("system", instructions), ("user", ex["question"])]).text
        grade = correctness(ex["question"], ex["answer"], reply)
        concise = concision(ex["answer"], reply)
        right += grade["correct"]
        short += concise
        print(f"{ex['question']:<20} correct={grade['correct']!s:<5} concise={concise!s:<5} ({len(reply)} chars)")
        if not grade["correct"]:
            print("   reply:", reply)
            print("   judge:", grade["explanation"])
    print(f"{name}: correctness {right}/{len(dataset)}, concision {short}/{len(dataset)}\n")

Reading the scores

  • Correctness: 3 of 5 for gpt-oss-20b and 4 of 5 for qwen3.8-27b. On these five data points the second model is one question ahead.
  • Concision: 2 of 5 and 1 of 5. Both models answered in one sentence of 81 to 130 characters, and the reference answers are short phrases, so most replies are longer than twice the reference and fail the rule. The metric measures length. It says nothing about whether the answer is good.
  • The failures carry the information. The judge faulted gpt-oss-20b on Google because the reply calls Google a technology company with internet services and leaves out search, which the reference names. It faulted the Mistral reply for calling Mistral "a family of large language models", which mixes up the model family with the company. It faulted qwen3.8-27b on LangSmith for describing "developing, testing, and monitoring" where the reference says "observing and evaluating".
  • Two of the three failures turn on wording. The replies are true, but a short reference sets a strict bar. The fix is in the data: a fuller reference answer, several accepted answers, or a judge prompt that says what counts as agreement.
  • The apps never see the reference answers. Only the judge does, so a model cannot copy the expected answer.

Rule metric vs judge metric vs human review

Rule in codeLLM as a judgeHuman review
CostNoneOne model call per answerSlow and the most expensive
RepeatableSame result every timeVaries a little between runsVaries between people
Good forLength, format, a required word, valid JSONMeaning: is the answer correct, on topic, groundedDeciding what "good" means and checking the judge
Weak spotCannot read meaningCan misjudge, and has its own habitsDoes not scale to every change

Where you use evaluation

  • Picking a model. Run the same data points through each candidate, as above, before paying for the larger one.
  • Changing a prompt. Re-run the data points after each edit and compare the table with the one before.
  • Checking a RAG app. Evaluating RAG adds metrics that look at the retrieved documents as well as the answer.
  • Choosing a tool. LangSmith (what the video uses) keeps datasets and experiments in a hosted interface and needs an account. DeepEval and RAGAS are open-source libraries with ready-made metrics that call a judge model of your choice. Langfuse keeps traces, datasets and experiments in one place.
Watch out. A score from five questions is a signal, not a ranking. One question moves a model from 4/5 to 3/5, a judge can flip its verdict on a borderline answer when the loop is run again, and a judge from the same family as a model under test can favor it (gpt-oss-120b grades gpt-oss-20b here). Grow the data points, read the explanation behind every failure, and repeat the run before trusting a gap between two models.
Try it yourself
  • Replace the reference answer for "What is Google?" with "A technology company known for its search engine and cloud services", re-run, and read how the judge's explanations change.
  • Add "groq:openai/gpt-oss-120b" to the list of models under test and compare its row with the other two. The judge is then grading its own model's answers: note what that does to your trust in its score.
  • Change the concision rule to len(response) < 4 * len(reference) and see which rows change.

Every expert started right here.