Threshold and strict_mode: what counts as a pass

Lesson 9 got a score back. Whether that score passes is a separate decision, and it is yours: the threshold.

python

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams
from pretend_judge import PretendJudge

params = [SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT]
case = LLMTestCase(input="Where is order A17?", actual_output="Order A17 is on its way.",
                   expected_output="Order A17 shipped on 3 March.")

One test case and one list of fields, reused by every snippet below.

To make this lesson about pass marks rather than judging, the judge is scripted: PretendJudge(score=7) answers 7 to every scoring question. That is the branch in g_eval_score from lesson 9 that checks judge.score.

Example

metric = GEval(name="Correctness", criteria="Gives the shipping news.",
               evaluation_params=params, model=PretendJudge(score=7))
metric.measure(case)
print(metric.score, metric.threshold, metric.is_successful())

0.7 against the default threshold of 0.5. It passes.

Example

metric = GEval(name="Correctness", criteria="Gives the shipping news.",
               evaluation_params=params, model=PretendJudge(score=7), threshold=0.8)
metric.measure(case)
print(metric.score, metric.threshold, metric.is_successful())

The same score against a threshold of 0.8 fails. Nothing about the answer or the judge changed, only what you were willing to accept.

Example

metric = GEval(name="Correctness", criteria="Gives the shipping news.",
               evaluation_params=params, model=PretendJudge(score=7), threshold=None)
metric.measure(case)
print(metric.score, metric.threshold, metric.is_successful())

A threshold of None keeps the score and gives no verdict at all. The docs call this score-only mode: useful for a metric you want to watch without letting it fail anything.

strict_mode: all or nothing

Example

right = LLMTestCase(input="Where is order A17?", actual_output="Order A17 shipped on 3 March.",
                    expected_output="Order A17 shipped on 3 March.")
strict = GEval(name="Correctness", criteria="Gives the shipping news.",
               evaluation_params=params, model=PretendJudge(), strict_mode=True)
for test in [right, case]:
    strict.measure(test)
    print(strict.score, strict.threshold, strict.is_successful(), strict.reason)

strict_mode=True changes two things. The threshold becomes 1. And G-Eval asks the judge for a score of exactly 1 or 0 instead of 0 to 10, which is the first branch in g_eval_score: 1 only when everything expected is there.

The right answer gets 1 and passes. The wrong one gets 0 and fails, with the same reason as before.

Choosing a threshold

Score a few answers you know are good and a few you know are bad, and put the threshold between them. A threshold picked before you have seen any scores is a guess.

Try it yourself

Script the judge with score=5 and find the highest threshold that still passes.
Create a strict metric with PretendJudge(score=7). Which wins, the script or strict mode?

PreviousG-Eval: criteria in plain words

Every expert started right here.