Evals: scoring the agent on many tickets

A test checks one path passes. An eval runs the agent on a set of real tickets and scores every answer, so you can see how good it is and whether a change helped.

Ticket is the model from lesson 6, and shop_model the stand-in:

from pydantic_ai import Agent
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected, Evaluator, EvaluatorContext

from shop_model import shop_model

agent = Agent(shop_model, output_type=Ticket)

async def sort(ticket: str) -> str:
    result = await agent.run(ticket)
    return result.output.category

An eval needs a task: an async function from input to output. sort runs the agent and returns the category.

class Urgent(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> bool:
        return ctx.output != "other" or "urgent" not in ctx.inputs.lower()

dataset = Dataset(
    name="ticket sorting",
    cases=[
        Case(name="double charge", inputs="I was charged twice", expected_output="billing"),
        Case(name="late parcel", inputs="My parcel never arrived", expected_output="shipping"),
        Case(name="money back", inputs="I want my money back", expected_output="billing"),
        Case(name="urgent broken", inputs="URGENT: the lamp came broken", expected_output="shipping"),
    ],
    evaluators=[EqualsExpected(), Urgent()],
)

• A Case is one input, with the output you expect.
• EqualsExpected checks the output equals expected_output.
• An evaluator you write is a class with evaluate. Urgent fails when a ticket marked urgent is sorted as other. ctx has inputs, output and expected_output.

report = dataset.evaluate_sync(sort, progress=False)
for case in report.cases:
    failed = [name for name, result in case.assertions.items() if not result.value]
    print(f"{case.name:15} {case.output:9} failed: {failed}")
print(f"passed: {report.averages().assertions:.1%}")

evaluate_sync runs the task on every case, then every evaluator on every output. case.assertions maps each evaluator's name to its result. The stand-in sorted the first two right. "I want my money back" has none of its keywords, so it came back other. "URGENT: the lamp came broken" failed both checks. Five of the eight checks passed: 62.5%.

In a terminal, report.print() draws the same results as a table. The loop above is how code reads them, for example to fail a build when the pass rate drops. Now add "money" and "broken" to sort_ticket, run the eval again, and check that no case that passed before now fails.

Evaluators that need a model

Some qualities have no exact answer: is the reply polite? LLMJudge(rubric="...") from pydantic_evals.evaluators asks a model to decide, and needs a real model to do it. The DeepEval and Ragas courses go deeper into judges and their pitfalls.