Real LLM judge: using GPT instead of the stand-in

Every score in this course came from a judge that compares words. That was the point: the whole course runs with no account. Here is the line that swaps it for a real model.

from pretend_judge import PretendJudge

relevancy = AnswerRelevancyMetric(model=PretendJudge())

relevancy = AnswerRelevancyMetric(model="gpt-4.1")

That is the change. The model argument takes an OpenAI model name, and DeepEval builds the judge for you. The metric, the test case, the threshold and the report are all unchanged.

export OPENAI_API_KEY="sk-..."

The key goes in the environment, not in your code. DeepEval also reads a .env file if there is one.

What changes when the judge is real

• Scores stop being repeatable. The same test case can score 0.8 and then 0.7. Thresholds need a little room, and a metric that flips around is a metric to look at again.
• It reads meaning. The refund answer that scored 0.33 in lesson 18 scores properly, because a real judge understands that the answer answers the question.
• It costs time and money. Every metric is one or more model calls per test case. The cache, -c from lesson 20, stops you paying twice for the same thing.
• Your criteria matter more. A vague G-Eval criterion gives a vague score, and the evaluation_steps from lesson 9 are how you tighten it.

Other providers

The same argument takes any model wrapped as a DeepEvalBaseLLM, which is exactly what you built in lesson 8. DeepEval ships wrappers for Azure OpenAI, Anthropic, Gemini, Ollama and LiteLLM, so a local model through Ollama is a reasonable middle ground: no bill, and a judge that reads meaning.

Keep the stand-in judge in the repository even so. It is the fastest way to check that a new metric is wired up correctly before you spend anything on it.