Custom metrics: a metric in plain Python

Some checks do not need a judge. Does the answer name the order the customer asked about? That is a line of Python, and BaseMetric lets it run in the same test run as G-Eval.

python

import re
from deepeval.metrics import BaseMetric


class MentionsOrderId(BaseMetric):
    def __init__(self, threshold=1.0):
        self.threshold = threshold

    def measure(self, test_case):
        asked = re.findall(r"\b[A-Z]\d+\b", test_case.input)
        self.score = 1.0 if asked and asked[0] in test_case.actual_output else 0.0
        self.reason = f"Looked for {asked[0] if asked else 'an order id'} in the answer."
        self.success = self.score >= self.threshold
        return self.score

A custom metric inherits BaseMetric and sets its own threshold. Its measure must set self.score and self.success; the reason is optional. This one finds the order id in the question and looks for it in the answer.

python

    async def a_measure(self, test_case):
        return self.measure(test_case)

    def is_successful(self):
        return self.success

    @property
    def __name__(self):
        return "Mentions order id"

Three more pieces the docs ask for. a_measure is the asynchronous version, and reusing measure is fine when nothing inside needs to wait. is_successful reports the result. __name__ is the name shown in reports.

Example

from deepeval.test_case import LLMTestCase

metric = MentionsOrderId()
for answer in ["Order A17 shipped on 3 March.", "Your order shipped on 3 March."]:
    metric.measure(LLMTestCase(input="Where is order A17?", actual_output=answer))
    print(metric.score, metric.is_successful(), metric.reason)

The second answer is right about the date and never says which order, so this metric fails it.

Next to a judged metric

Example

from deepeval import evaluate
from deepeval.evaluate import DisplayConfig
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams
from pretend_judge import PretendJudge

correctness = GEval(name="Correctness", criteria="Gives the shipping news.", model=PretendJudge(),
                    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT])
case = LLMTestCase(input="Where is order A17?", actual_output="Your order shipped on 3 March.",
                   expected_output="Order A17 shipped on 3 March.")
result = evaluate(test_cases=[case], metrics=[MentionsOrderId(), correctness],
                  display_config=DisplayConfig(print_results=False, show_indicator=False))
for data in result.test_results[0].metrics_data:
    print(data.name, data.score, data.success)

One test case, two metrics, two different verdicts. G-Eval passes the answer at 0.7, because it found most of the expected words. The custom metric fails it, because the one word it cares about is missing. The test case fails, because a test case needs every metric to pass.

That combination is the usual shape: a judged metric for whether the answer is good, and a few plain checks for things that must always be true.

Try it yourself

Make the metric score 0.5 when the answer names a different order than the one asked about.
Remove the __name__ property and see what name the results use instead.

PreviousDAGMetric: a decision tree judge

You understood something today that you didn't yesterday.