Evaluating LLM apps
LLM evaluation is the practice of running an LLM app on a fixed set of test cases and scoring its answers, so that a change of model, prompt or retriever is judged by numbers instead of by feel.
Last updated: 08 Oct, 2026 · LangChain 1.4
An LLM app answers differently each time it is asked, and a prompt change that fixes one question can break three others. An evaluation turns "it seems better" into a table that can be compared from one run to the next. This lesson builds one as a plain Python loop with a judge model, which needs no account beyond the keys already set up.
Choosing between models with data
The video starts from a simple chatbot and the questions that come up once it exists. Which model should it use? Cost is one factor, but accuracy on the specific use case matters more. How do you decide that an output is good enough? That needs some ground truth: for a given input, the output you expect. And who does the comparing? The video's answer is LLM as a judge: a second model, given a prompt, that compares each response with the ground truth and gives a verdict.
The video lays the work out in four steps:
- Gather data points. Each one is an input and the output expected for it.
- Build the judge. A prompt that tells an LLM how to grade a response against the expected output.
- Define the metrics. What each score means: for example, correct or not, concise or not.
- Compare models. Run the same data points through each candidate and keep the one with the best metrics.
Writing the data points
A data point is a question with a short reference answer. These are the five from the video: five questions about well-known tools and companies, each answered in one phrase.
dataset = [
{"question": "What is LangChain?", "answer": "A framework for building LLM applications"},
{"question": "What is LangSmith?", "answer": "A platform for observing and evaluating LLM applications"},
{"question": "What is OpenAI?", "answer": "A company that creates Large Language Models"},
{"question": "What is Google?", "answer": "A technology company known for search"},
{"question": "What is Mistral?", "answer": "A company that creates Large Language Models"},
]Writing a judge and a metric
The grade schema and the judge model
The judge returns structured output (the Structured output lesson covers it). The schema has two fields in a deliberate order: explanation first, then correct. The model writes its reasoning before it commits to a verdict, and the verdict arrives as a real True or False, so the code compares nothing against a string. Temperature 0 keeps the grading as steady as a model allows.
from typing import Annotated
from typing_extensions import TypedDict
from langchain.chat_models import init_chat_model
class Grade(TypedDict):
explanation: Annotated[str, ..., "Explain your reasoning for the grade"]
correct: Annotated[bool, ..., "True if the predicted answer agrees with the real answer"]
judge = init_chat_model("groq:openai/gpt-oss-120b", temperature=0).with_structured_output(Grade)The correctness function
The function builds the grading prompt from the question, the reference answer and the response, then asks the judge. The system message is the video's: an expert professor grading students' answers.
def correctness(question, reference, response):
prompt = (f"You are grading the following question:\n{question}\n"
f"Here is the real answer:\n{reference}\n"
f"You are grading the following predicted answer:\n{response}")
return judge.invoke([
("system", "You are an expert professor specialized in grading students' answers to questions."),
("user", prompt),
]) # a dict: {"explanation": ..., "correct": True or False}A rule-based metric
Not every metric needs a model. The video's second metric, concision, is a length check: the response must be shorter than twice the reference answer. It costs nothing and returns the same result every time, but it measures length and not quality.
def concision(reference, response):
return len(response) < 2 * len(reference) # shorter than twice the reference answerThe video runs this with LangSmith: its client.evaluate calls the app for every data point in a dataset stored on LangSmith, applies both metrics and shows each experiment on a results page. Its two models are OpenAI's gpt-4o-mini and gpt-4-turbo. The loop below does the same four steps in plain Python on Groq's models, and prints the table in the terminal, so no account is needed.
Running two models through the same data points
The two models under test are gpt-oss-20b and qwen3.8-27b, both served by Groq. The judge is a third model, gpt-oss-120b, so no model grades its own answers. Every response is graded on correctness by the judge and on concision by the rule. For a wrong verdict the loop also prints the reply and the judge's explanation, because the reason behind a failure is what tells you what to fix.
from typing import Annotated
from typing_extensions import TypedDict
from langchain.chat_models import init_chat_model
dataset = [
{"question": "What is LangChain?", "answer": "A framework for building LLM applications"},
{"question": "What is LangSmith?", "answer": "A platform for observing and evaluating LLM applications"},
{"question": "What is OpenAI?", "answer": "A company that creates Large Language Models"},
{"question": "What is Google?", "answer": "A technology company known for search"},
{"question": "What is Mistral?", "answer": "A company that creates Large Language Models"},
]
class Grade(TypedDict):
explanation: Annotated[str, ..., "Explain your reasoning for the grade"]
correct: Annotated[bool, ..., "True if the predicted answer agrees with the real answer"]
judge = init_chat_model("groq:openai/gpt-oss-120b", temperature=0).with_structured_output(Grade)
def correctness(question, reference, response):
prompt = (f"You are grading the following question:\n{question}\n"
f"Here is the real answer:\n{reference}\n"
f"You are grading the following predicted answer:\n{response}")
return judge.invoke([
("system", "You are an expert professor specialized in grading students' answers to questions."),
("user", prompt),
])
def concision(reference, response):
return len(response) < 2 * len(reference)
instructions = "Respond to the user's question in a short, concise manner (one short sentence)."
for name in ["groq:openai/gpt-oss-20b", "groq:qwen/qwen3.8-27b"]:
app = init_chat_model(name, temperature=0)
right = short = 0
for ex in dataset:
reply = app.invoke([("system", instructions), ("user", ex["question"])]).text
grade = correctness(ex["question"], ex["answer"], reply)
concise = concision(ex["answer"], reply)
right += grade["correct"]
short += concise
print(f"{ex['question']:<20} correct={grade['correct']!s:<5} concise={concise!s:<5} ({len(reply)} chars)")
if not grade["correct"]:
print(" reply:", reply)
print(" judge:", grade["explanation"])
print(f"{name}: correctness {right}/{len(dataset)}, concision {short}/{len(dataset)}\n")What is LangChain? correct=True concise=False (109 chars) What is LangSmith? correct=True concise=True (100 chars) What is OpenAI? correct=True concise=False (102 chars) What is Google? correct=False concise=False (104 chars) reply: Google is a multinational technology company that specializes in internet-related services and products. judge: The predicted answer correctly identifies Google as a technology company, but it omits the key detail that Google is specifically known for its search service. The real answer emphasizes the search aspect, which is not captured in the prediction, so the answer does not fully agree. What is Mistral? correct=False concise=True (82 chars) reply: Mistral is a family of large language models created by the AI startup Mistral AI. judge: The real answer states that Mistral is a company that creates large language models. The predicted answer describes Mistral as a family of large language models produced by the startup Mistral AI, which conflates the model family with the company itself. Therefore, the predicted answer does not correctly match the real answer. groq:openai/gpt-oss-20b: correctness 3/5, concision 2/5 What is LangChain? correct=True concise=False (86 chars) What is LangSmith? correct=False concise=True (81 chars) reply: LangSmith is a platform for developing, testing, and monitoring LLM applications. judge: The predicted answer correctly identifies LangSmith as a platform related to LLM applications, but it mischaracterizes its primary purpose. The real answer specifies that LangSmith is for *observing and evaluating* LLM applications. The prediction instead says it is for *developing, testing, and monitoring*, adding 'developing' which is not mentioned and substituting 'testing' for 'evaluating' and 'monitoring' for 'observing'. These differences constitute an inaccurate description, so the answer does not fully agree with the real answer. What is OpenAI? correct=True concise=False (130 chars) What is Google? correct=True concise=False (116 chars) What is Mistral? correct=True concise=False (88 chars) groq:qwen/qwen3.8-27b: correctness 4/5, concision 1/5
Reading the scores
- Correctness: 3 of 5 for gpt-oss-20b and 4 of 5 for qwen3.8-27b. On these five data points the second model is one question ahead.
- Concision: 2 of 5 and 1 of 5. Both models answered in one sentence of 81 to 130 characters, and the reference answers are short phrases, so most replies are longer than twice the reference and fail the rule. The metric measures length. It says nothing about whether the answer is good.
- The failures carry the information. The judge faulted gpt-oss-20b on Google because the reply calls Google a technology company with internet services and leaves out search, which the reference names. It faulted the Mistral reply for calling Mistral "a family of large language models", which mixes up the model family with the company. It faulted qwen3.8-27b on LangSmith for describing "developing, testing, and monitoring" where the reference says "observing and evaluating".
- Two of the three failures turn on wording. The replies are true, but a short reference sets a strict bar. The fix is in the data: a fuller reference answer, several accepted answers, or a judge prompt that says what counts as agreement.
- The apps never see the reference answers. Only the judge does, so a model cannot copy the expected answer.
Rule metric vs judge metric vs human review
| Rule in code | LLM as a judge | Human review | |
|---|---|---|---|
| Cost | None | One model call per answer | Slow and the most expensive |
| Repeatable | Same result every time | Varies a little between runs | Varies between people |
| Good for | Length, format, a required word, valid JSON | Meaning: is the answer correct, on topic, grounded | Deciding what "good" means and checking the judge |
| Weak spot | Cannot read meaning | Can misjudge, and has its own habits | Does not scale to every change |
Where you use evaluation
- Picking a model. Run the same data points through each candidate, as above, before paying for the larger one.
- Changing a prompt. Re-run the data points after each edit and compare the table with the one before.
- Checking a RAG app. Evaluating RAG adds metrics that look at the retrieved documents as well as the answer.
- Choosing a tool. LangSmith (what the video uses) keeps datasets and experiments in a hosted interface and needs an account. DeepEval and RAGAS are open-source libraries with ready-made metrics that call a judge model of your choice. Langfuse keeps traces, datasets and experiments in one place.
Related
- Previous: Fine-tuning on custom data
- Next: LLM gateways
- See also: Evaluating RAG and Guardrails
- Reference: LangSmith evaluation docs
- Go deeper: Evals & Guardrails Interview Questions
- Replace the reference answer for "What is Google?" with "A technology company known for its search engine and cloud services", re-run, and read how the judge's explanations change.
- Add
"groq:openai/gpt-oss-120b"to the list of models under test and compare its row with the other two. The judge is then grading its own model's answers: note what that does to your trust in its score. - Change the concision rule to
len(response) < 4 * len(reference)and see which rows change.
Every expert started right here.