What you are going to build

DeepEval tests the answers an LLM app gives, the way unit tests check ordinary code. This course builds a test suite for one support bot, a piece at a time, and every piece runs on your machine with no API key.

An answer from a model is text, and it can be worded differently every time. You cannot compare it with one right string and call it tested. DeepEval gives you three pieces to do it properly: a test case that holds one question and one answer, a metric that scores it from 0 to 1, and a judge, usually another model, that the metric asks whenever a score needs reading rather than counting.

A metric, working

Here is the kind of check the course builds: one metric grading two answers to the same question. One gives the shipping news, the other does not.

python

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams
from pretend_judge import PretendJudge

correctness = GEval(
    name="Correctness",
    criteria="The answer gives the same shipping news as the expected answer.",
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
    model=PretendJudge(),
)

Example

for answer in ["Order A17 shipped on 3 March.", "Order A17 is on its way."]:
    case = LLMTestCase(input="Where is order A17?", actual_output=answer,
                       expected_output="Order A17 shipped on 3 March.")
    correctness.measure(case)
    print(correctness.score, correctness.is_successful(), correctness.reason)

The first answer scores 1.0 and passes. The second scores 0.3 and fails, and the reason names the two words the judge could not find in it. Every piece of that, the test case, the metric, the pass mark and the judge, is a lesson of its own.

The judge here is a stand-in you write yourself in lesson 8. DeepEval cannot tell it from a real model, which is why none of this needed a key.

What you will have built

Piece	What it does	Lesson
Test cases	One question, the answer your app gave, and the answer you hoped for	2
Metrics that count	A score with no model involved at all	3
A stand-in judge	Answers every question a metric asks, with no key	8
Metrics in your own words	G-Eval criteria, a decision tree, and plain Python	9 to 12
RAG metrics	Was the answer relevant, and faithful to what the bot read	13 to 15
Datasets	A file of questions that every run starts from	16
Traces	Scoring an agent's steps as well as its answer	17 to 19
A suite in CI	Evals that fail a build like any other test	20

What you need before you start

One install.

bash

pip install -U deepeval

	Needed for this course?	When you do need it
Python 3.9 or later	Yes	Now
An OpenAI API key	No	Lesson 21, when you swap in a real judge
A Confident AI account	No	Never, for this course. It is the optional hosted platform from the DeepEval team

Almost every DeepEval metric asks a model to judge, and by default that model is one of OpenAI's, which is why other tutorials start by setting a key. This one writes a stand-in judge instead. The metrics around it are the real ones, so the scores are real scores.

How this course is written

Every snippet on these pages was run against deepeval 4.2.3 in a real Python environment, and the output shown is what came back. Nothing is typed by hand.

Try it yourself

Install DeepEval now, so the next lessons are one command away.
Download pretend_judge.py into the folder you will work in.
Before you read on, guess what the second answer would need to say to score 1.0.

Back toAll frameworks

Every expert started right here.