G-Eval: criteria in plain words

Lesson 8 built a judge that could not answer anything. G-Eval is the metric that uses a judge in the plainest way: you describe a good answer in a sentence, and the judge scores against it.

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams
from pretend_judge import PretendJudge

correctness = GEval(
    name="Correctness",
    criteria="The answer gives the same shipping news as the expected answer.",
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT],
    model=PretendJudge(),
)

pretend_judge.py is the file from lesson 8, with every answer this course needs already in it. Each answer is shown in the lesson for its metric, and G-Eval's are further down this page.

criteria is the sentence. evaluation_params lists which fields of the test case the judge gets to see.

wrong = LLMTestCase(input="Where is order A17?", actual_output="Order A17 is on its way.",
                    expected_output="Order A17 shipped on 3 March.")
correctness.measure(wrong)
print(correctness.score, correctness.is_successful(), correctness.reason)

A wrong answer, passed with full marks. The criteria talk about the expected answer, but evaluation_params only sent the actual output, so the judge never saw what it was meant to compare against. The docs warn about exactly this: include every field the criteria mention.

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams
from pretend_judge import PretendJudge

correctness = GEval(
    name="Correctness",
    criteria="The answer gives the same shipping news as the expected answer.",
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
    model=PretendJudge(),
)

wrong = LLMTestCase(input="Where is order A17?", actual_output="Order A17 is on its way.",
                    expected_output="Order A17 shipped on 3 March.")
correctness.measure(wrong)
print(correctness.score, correctness.is_successful(), correctness.reason)

Now it fails, with 0.3, and the reason names what was missing. Same metric, same judge, one more field in the list.

What the judge was asked

G-Eval asks its judge two things. First it hands over the criteria and asks for evaluation steps.

@answers("g_eval.Steps")
def g_eval_steps(prompt, judge):
    return {"steps": ["Compare the actual output with the expected output."]}

Then it sends those steps with the test case fields and asks for a score from 0 to 10 and a reason. This judge's rule is the share of the expected answer's words found in the actual one. The two branches at the top belong to lesson 10.

@answers("g_eval.ReasonScore")
def g_eval_score(prompt, judge):
    actual = section(prompt, "Actual Output")
    expected = section(prompt, "Expected Output")
    found = share(expected, actual)
    if "STRICTLY EITHER 1" in prompt:
        score = 1 if found == 1 else 0
    elif judge.score is not None:
        score = judge.score
    else:
        score = round(10 * found)
    missing = sorted(words(expected) - words(actual))
    return {"score": score, "reason": "Missing: " + ", ".join(missing) if missing else "Covers everything expected."}

G-Eval turns the 0 to 10 answer into a score from 0 to 1, which is how a 3 became 0.3.

correctness.verbose_mode = True
correctness.measure(wrong)

Verbose mode from lesson 4, on a judged metric. The log shows the steps the judge wrote, then the score and the reason.

evaluation_steps: writing the steps yourself

Instead of criteria you can give the steps directly. G-Eval then skips the first question and scores with yours, which makes a metric more predictable. You give one or the other, not both.

checked = GEval(
    name="Correctness",
    evaluation_steps=["Check that the actual output gives the same date as the expected output."],
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
    model=PretendJudge(),
    verbose_mode=True,
)
checked.measure(wrong)

The steps in the log are the ones you wrote. Only one question reached the judge this time: the score.