ErrorConfig: when a metric cannot run

Every run so far had what its metric needed. Real test sets are messier, and one test case a metric cannot score stops the whole run unless you say otherwise.

broken = cases + [
    LLMTestCase(input="Can I get a refund?", actual_output="Yes, within 30 days."),
]

A third question, about refunds, with no expected answer written yet. Exact match needs one.

from deepeval import evaluate
from deepeval.evaluate import DisplayConfig

quiet = DisplayConfig(print_results=False, show_indicator=False)
try:
    evaluate(test_cases=broken, metrics=[ExactMatchMetric()], display_config=quiet)
except Exception as error:
    print(type(error).__name__)
    print(error)

The whole run stopped. The two test cases that were fine got no result either, because evaluate raised instead of returning. The message says exactly what is missing and which metric wanted it.

Two ways to carry on

ErrorConfig decides what a run does when a metric cannot finish. It has two switches, and they mean different things.

def show(result):
    for test in result.test_results:
        errors = [data.error for data in test.metrics_data or []]
        print(test.name, test.success, errors)

A small helper, so the next two runs print the same way: each test case's name, whether it passed, and any error its metrics recorded.

from deepeval.evaluate import ErrorConfig

result = evaluate(test_cases=broken, metrics=[ExactMatchMetric()], display_config=quiet,
                  error_config=ErrorConfig(skip_on_missing_params=True))
show(result)

skip_on_missing_params skips a metric on any test case that lacks what the metric needs. The refund question has no metric data at all, and it is reported as a success.

Look at the summary too. It counts two tests, not three. The skipped case is in test_results but left out of the pass rate, so the two disagree about how many tests there were.

result = evaluate(test_cases=broken, metrics=[ExactMatchMetric()], display_config=quiet,
                  error_config=ErrorConfig(ignore_errors=True))
show(result)

ignore_errors lets the metric fail, records the error on the test case, and carries on. The refund question now fails, with the reason attached, and the summary counts three tests.