DeepEval overview
DeepEval is an open-source Python framework for testing LLM applications: a test case holds one input and what the app answered, and a metric scores that answer from 0 to 1, most often by asking a second LLM to act as the judge.
Last updated: 05 Oct, 2026 · DeepEval 4.2.8
DeepEval is developed on GitHub by Confident AI and released under the Apache 2.0 licence. It works the way pytest works for ordinary code: you write test cases, pick metrics with a pass mark, and a run tells you which cases passed and why the others failed. This course uses version 4.2.8.
An LLM app gives a different answer every time it is asked, in different words, so a plain assert answer == expected fails good answers and passes bad ones. DeepEval replaces that assert with metrics that read the answer the way a person would: is it correct, is it on topic, is it backed by the documents the app retrieved, did the agent call the right tool.
The video starts from a simple chatbot: an input goes in, an output comes out. Then come the questions every team asks. Which LLM should the app use? Cost is one factor, but more important is how accurate the output is for this use case. And how do you decide an output is good? You compare it with a ground truth, so you need data in the form "for this input, this should be the output".
From there the video lays out four steps: gather data points of input and expected output, use an LLM as a judge, apply evaluation metrics, and compare several LLMs to pick the one with the best scores. The video runs those steps in LangSmith; this course runs every one of them with DeepEval, in plain Python files on your machine.
Evaluating the TechNest support bot
Every lesson tests the same app: a support bot for TechNest, an online electronics store. It starts as a small RAG app and grows into an agent with tools and a chat that remembers the conversation.
- The app. A catalog of 15 products, policies and FAQ entries, a search that picks the three closest entries, and a Groq model that writes the answer from them.
- The goldens. Five questions a customer might ask, each with the answer TechNest expects.
- The judge. A second Groq model, wrapped so DeepEval can send it every metric's questions.
- The metrics. Correctness in your own words with G-Eval, the five RAG metrics, tool correctness and task completion for the agent, and conversation metrics for the chat.
- The suite. A test file that
deepeval test runruns in CI, and a final project that scores the bot and catches a regression.
Test cases, metrics, a judge and datasets
These are the DeepEval names you will meet, in the order the course introduces them.
| DeepEval piece | What it holds or does | Taught in |
|---|---|---|
LLMTestCase | One input, the app's answer, and optionally the expected answer and retrieved chunks | LLMTestCase |
| Metrics with no model | Plain Python scores such as exact match and a regex pattern | ExactMatchMetric and PatternMatchMetric |
assert_test and deepeval test run | Metrics as pytest tests that pass or fail | First test run |
DeepEvalBaseLLM | The base class that turns any model into a judge | Custom judge model |
GEval | A metric written as criteria in plain words | G-Eval |
| RAG metrics | Faithfulness, answer relevancy, contextual precision, recall and relevancy | RAG metrics |
EvaluationDataset and evaluate() | Goldens in a file, and one run of every metric over every case | Datasets and goldens, evaluate() |
@observe | Tracing, so an agent's steps can be scored | LLM tracing |
ConversationalTestCase | A whole chat, turn by turn | Multi-turn evaluation |
Keys you need
Every judged score in this course comes from a real model, and both the app and the judge run on Groq's free tier, so one key is enough. It needs no card.
| Key | Used for | Where to get it |
|---|---|---|
GROQ_API_KEY | The app's answers and the judge | console.groq.com/keys |
GOOGLE_API_KEY | Optional: Gemini as a second judge, shown in Integrations | aistudio.google.com/apikey |
Related
- Next: Installation and setup
- Reference: DeepEval documentation
Every expert started right here.