Structured output: checking every answer

A program that uses a model's answer has to check it every time. Pydantic from Python for AI catches answers in the wrong shape; it cannot catch answers that are wrong.

This lesson reuses reply, the tickets, both prompts from lesson 10 and examples from lesson 11.

from typing import Literal

from pydantic import BaseModel, Field, ValidationError


class Triage(BaseModel):
    category: Literal["billing", "shipping", "other"]
    priority: int = Field(ge=1, le=5)

def check(answer):
    try:
        return Triage.model_validate_json(answer)
    except ValidationError:
        return None

check returns a Triage for an answer in the right shape and None for anything else.

prompts = {"vague": [{"role": "system", "content": vague}], "few-shot": [{"role": "system", "content": structured}, *examples]}

for label, start_messages in prompts.items():
    results = [check(reply(start_messages + [{"role": "user", "content": text}])) for text, _ in tickets]
    valid = sum(result is not None for result in results)
    print(f"{label}: {valid} of {len(results)} answers passed the check")

sum over True and False counts the Trues. None of the vague prompt's answers pass. The few-shot prompt's all pass, including the password ticket filed as billing: Pydantic checked that billing is an allowed value, and it is.

Constrained generation

Many hosted APIs offer a structured output mode: you send a JSON Schema, like the one Pydantic writes, and the provider restricts which tokens the model may pick so the answer always fits the schema. It removes shape failures entirely. It does not remove wrong answers, so the check in the next lesson is still needed.