evaluate(): many test cases in one run

Part 1 measured one test case at a time. evaluate takes a list of test cases and a list of metrics, runs every metric on every case, and reports on all of them together.

from deepeval.metrics import ExactMatchMetric
from deepeval.test_case import LLMTestCase

cases = [
    LLMTestCase(input="Where is order A17?",
                actual_output="Order A17 shipped on 3 March.",
                expected_output="Order A17 shipped on 3 March."),
    LLMTestCase(input="Where is order B22?",
                actual_output="Order B22 is on its way.",
                expected_output="Order B22 shipped on 5 March."),
]

Two questions this time. The bot gets order A17 exactly right and words its answer about B22 differently from the expected one.

from deepeval import evaluate

evaluate(test_cases=cases, metrics=[ExactMatchMetric()])

That is a lot of output for two test cases, and all of it is useful once you know where to look.

The first line names the metric that is running and the settings it was created with.

One box per test case. A passing case gets a single line. A failing case shows its input, both outputs, and each metric's score, threshold and reason, which is everything you need to see why.

Aggregate Metrics averages each metric over every test case: 0.50 here, one pass and one fail.

The summary gives the pass rate for the whole run. DeepEval calls one run of evaluate a test run.

The notices at the end are about hyperparameters and about Confident AI, the hosted platform from the DeepEval team. Both are optional, and nothing in this course needs either.

Two things you did not see

The metrics ran concurrently rather than one after another, which is the default. And the run was saved: DeepEval writes the latest test run into a folder called .deepeval in the directory you ran from, so do not be surprised to find it there.