deepeval test run: evals as unit tests
Everything so far printed scores for a person to read. A build server cannot read. It needs a command that comes back angry when the bot got worse, and that is what this one does.
Three files, in one folder. The bot, the judge you wrote in lesson 8, and a test file.
POLICY = [
"Order A17 shipped on 3 March by courier.",
"Refunds are paid within 5 working days.",
]
def retrieve(question):
asked = set(question.lower().strip("?").split())
return [doc for doc in POLICY if asked & set(doc.lower().strip(".").split())]
def support_bot(question):
documents = retrieve(question)
answer = documents[0] if documents else "Sorry, I could not find that."
return answer, documentssupport_bot.py, unchanged from lesson 16.
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from deepeval.test_case import LLMTestCase
from pretend_judge import PretendJudge
from support_bot import support_bot
def case(question):
answer, documents = support_bot(question)
return LLMTestCase(input=question, actual_output=answer, retrieval_context=documents)test_support_bot.py starts by importing the bot, and turning a question into a test case the same way lesson 16 did. assert_test is the new piece: it takes a test case and metrics, and raises when any metric fails.
def test_order_question():
assert_test(case("Where is order A17?"),
[AnswerRelevancyMetric(model=PretendJudge()), FaithfulnessMetric(model=PretendJudge())])
def test_unknown_order():
assert_test(case("Where is order Z99?"),
[AnswerRelevancyMetric(model=PretendJudge()), FaithfulnessMetric(model=PretendJudge())])Two tests, each a plain function whose name starts with test_. The first asks about an order the policy covers. The second asks about one it does not.
deepeval test run test_support_bot.py -d failing --tb=no -q.
FRunning teardown with pytest sessionfinish...
============================= slowest 10 durations =============================
0.01s call test_support_bot.py::test_order_question
0.01s call test_support_bot.py::test_unknown_order
(4 durations < 0.005s hidden. Use -vv to show these durations.)
=========================== short test summary info ============================
FAILED test_support_bot.py::test_unknown_order - AssertionError: Metrics: Answer Relevancy (score: 0.0, threshold: 0.5, stri...
Test Results
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ ┃ ┃ ┃ ┃ Overall Success ┃
┃ Test case ┃ Metric ┃ Score ┃ Status ┃ Rate ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ test_unknown_… │ │ │ │ 50.0% | │
│ │ │ │ │ passed=1 | │
│ │ │ │ │ failed=1 │
│ │ Answer │ 0.0 │ FAILED │ │
│ │ Relevancy │ (threshold=0.… │ │ │
│ │ │ evaluation │ │ │
│ │ │ model=pretend │ │ │
│ │ │ judge, │ │ │
│ │ │ reason=The │ │ │
│ │ │ score is │ │ │
│ │ │ 0.00., │ │ │
│ │ │ error=None) │ │ │
│ │ Faithfulness │ 1.0 │ PASSED │ │
│ │ │ (threshold=0.… │ │ │
│ │ │ evaluation │ │ │
│ │ │ model=pretend │ │ │
│ │ │ judge, │ │ │
│ │ │ reason=The │ │ │
│ │ │ score is │ │ │
│ │ │ 1.00., │ │ │
│ │ │ error=None) │ │ │
│ Note: Use │ │ │ │ │
│ Confident AI │ │ │ │ │
│ with DeepEval │ │ │ │ │
│ to analyze │ │ │ │ │
│ failed test │ │ │ │ │
│ cases for more │ │ │ │ │
│ details │ │ │ │ │
└────────────────┴─────────────────┴────────────────┴────────┴─────────────────┘
⚠ WARNING: No hyperparameters logged.
» Log hyperparameters to attribute prompts and models to your test runs.
================================================================================
✓ Evaluation completed 🎉! (time taken: 0.16s | token cost: None)
» Test Results (2 total tests):
» Pass Rate: 50.0% | Passed: 1 | Failed: 1
===============================================================================
=
» Want to share evals with your team, or a place for your test cases to live? ❤️
🏡
» Run 'deepeval view' to analyze and save testing results on Confident AI.One passed, one failed. The failure is the Z99 question: the retriever matches on any shared word, so it handed back the line about order A17, and answer relevancy scored the answer 0 against the question that was asked.
The table is the same report as evaluate prints, with the metrics for each test case, their scores and their reasons. -d failing keeps the passing cases out of it.
The command exits non-zero when a test fails, which is the whole point: a pull request that makes the bot worse turns the build red without anybody reading a score.
The flags worth knowing
| Flag | What it does |
|---|---|
-n 4 | Run test cases in parallel, across four processes |
-c | Use cached results for test cases and metrics that have not changed |
-i | Ignore metric errors instead of stopping, like ErrorConfig from lesson 7 |
-s | Skip test cases missing a field a metric needs |
-v | Turn on verbose mode for every metric, as in lesson 4 |
-d failing | Show only the failing test cases in the report |
In a pipeline
- name: Run evals
run: |
pip install -U deepeval
deepeval test run test_support_bot.pyOne step in any CI system that can run a shell command. Nothing about the test file changes.
deepeval test run passes pytest's own flags through, which is where --tb=no and -q above came from. The docs ask you not to call pytest directly: the test run, the report and the caching all come from DeepEval's own command.- Fix the retriever so an unknown order returns nothing, and run the command again.
- Add
-n 2and compare how long the run takes.
Every expert started right here.