ExactMatchMetric: your first metric

A test case holds an answer and a metric scores it. The simplest metric in DeepEval does what lesson 1 did by hand, and it comes first because it needs no judge at all.

from deepeval.metrics import ExactMatchMetric

metric = ExactMatchMetric()

metric.measure(case)

print("score:  ", metric.score)
print("reason: ", metric.reason)
print("success:", metric.is_successful())

measure scores the test case and keeps the result on the metric object, which is why the next three lines read it from metric.

score runs from 0 to 1. Every metric in DeepEval scores on that same scale.

reason says why, in a sentence.

is_successful() says whether the score was good enough to pass.

The reworded answer

reworded = LLMTestCase(
    input="Where is order A17?",
    actual_output="Your order A17 was shipped on 3 March.",
    expected_output="Order A17 shipped on 3 March.",
)
metric.measure(reworded)

print("score:    ", metric.score)
print("reason:   ", metric.reason)
print("success:  ", metric.is_successful())
print("threshold:", metric.threshold)

Score 0, which is lesson 1's first problem again: the meaning is right and the characters are not.

The last line is the pass mark. threshold is the lowest score that counts as a pass, and a metric succeeds when its score reaches it. For exact match the default is 1, so only a perfect match passes. Most other metrics default to 0.5, which starts to matter in lesson 10.

Measuring the second test case replaced the first result. A metric object remembers only the last thing it measured.