Measuring a prompt: a small evaluation

Reading answers by eye said few-shot was better. An evaluation makes that a number: run each prompt on tickets with known answers, and count.

This lesson reuses reply, the tickets, the prompts, examples and check from lessons 9 to 12.

def score(start_messages):
    right = 0
    for text, expected in tickets:
        result = check(reply(start_messages + [{"role": "user", "content": text}]))
        if result is not None and result.category == expected:
            right += 1
    return right / len(tickets)

An answer scores only if it passes the check and has the right category. The result is the fraction right. The labelled tickets are a tiny test set: cases with known answers, used only for measuring, never pasted into a prompt.

candidates = {
    "vague": [{"role": "system", "content": vague}],
    "structured": [{"role": "system", "content": structured}],
    "few-shot": [{"role": "system", "content": structured}, *examples],
}
for label, start_messages in candidates.items():
    print(f"{label:10} {score(start_messages):.0%}")

:.0% shows a fraction as a percentage. Three prompts, three numbers, one clear winner.

Five tickets is far too few. One ticket is 20 points here, so a single lucky answer moves the score a lot. A real test set has dozens to hundreds of cases, taken from real tickets, including the awkward ones. The idea is the same at any size.

What the number is for

Every change you try from now on, a new example, a different wording, a bigger model, gets the same test. A change that feels better and scores worse is worse. Frameworks like DeepEval, Ragas and Promptfoo, each with its own course, are this loop with more metrics and reports.