The ticket sorter, measured
Lesson 0 promised a ticket-sorting prompt, a score for how often it is right, and its cost and speed. One script reports all three, on the real model.
One improvement goes in first. Lesson 11 left the password ticket wrong, and its exercise suggested an example about logins. The prompt file adds exactly that, a fourth example, and the score decides whether it helped.
The prompt, in its own file
from typing import Literal
from pydantic import BaseModel, Field, ValidationError
class Triage(BaseModel):
category: Literal["billing", "shipping", "other"]
priority: int = Field(ge=1, le=5)
def check(answer):
try:
return Triage.model_validate_json(answer)
except ValidationError:
return None
SYSTEM = """You sort customer support tickets for an online shop.
Categories:
- billing: payments, charges, refunds
- shipping: parcels, delivery, damaged boxes
- other: anything else
Reply with one line of JSON and nothing else, like this:
{"category": "shipping", "priority": 3}
priority is 1 (can wait) to 5 (urgent)."""
EXAMPLES = [
{"role": "user", "content": "You took money from my card two times"},
{"role": "assistant", "content": '{"category": "billing", "priority": 4}'},
{"role": "user", "content": "Where is my delivery? It is a week late"},
{"role": "assistant", "content": '{"category": "shipping", "priority": 3}'},
{"role": "user", "content": "Can I change the email on my account?"},
{"role": "assistant", "content": '{"category": "other", "priority": 2}'},
{"role": "user", "content": "I forgot my login details"},
{"role": "assistant", "content": '{"category": "other", "priority": 2}'},
]
TICKETS = [
("I was charged twice for one order", "billing"),
("My parcel has not arrived", "shipping"),
("Can I get a refund for the blue mug?", "billing"),
("How do I change my password?", "other"),
("The parcel arrived but the box was crushed", "shipping"),
]The Triage model and check from lesson 12, the structured system prompt from lesson 10, the examples from lesson 11 plus the new login example, and the labelled tickets. Keeping the prompt apart from the code means a change to the prompt is a change to one file, easy to review and score.
The script
import time
from transformers import AutoModelForCausalLM, AutoTokenizer
from prompt import check, EXAMPLES, SYSTEM, TICKETS
name = "Qwen/Qwen2.5-0.5B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)def sort_ticket(text):
messages = [{"role": "system", "content": SYSTEM}, *EXAMPLES, {"role": "user", "content": text}]
ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt", return_dict=True)
output = model.generate(**ids, max_new_tokens=30, do_sample=False)
new = output[0][ids["input_ids"].shape[1]:]
return tokenizer.decode(new, skip_special_tokens=True), ids["input_ids"].shape[1], len(new)return_dict=True gives the ids and the attention mask in one dictionary, ready for generate. The function returns the answer and both token counts, which the cost line needs.
right, tokens_in, tokens_out = 0, 0, 0
started = time.perf_counter()
for text, expected in TICKETS:
answer, used_in, used_out = sort_ticket(text)
tokens_in, tokens_out = tokens_in + used_in, tokens_out + used_out
result = check(answer)
ok = result is not None and result.category == expected
right += ok
print(f"{'ok ' if ok else 'BAD'} {expected:9} {answer}")
seconds = time.perf_counter() - started
print(f"accuracy {right}/{len(TICKETS)}")
print(f"tokens {tokens_in} in, {tokens_out} out")
print(f"cost ${(tokens_in * 0.50 + tokens_out * 1.50) / 1_000_000:.6f} at the example prices")
print(f"time {seconds / len(TICKETS):.2f} seconds a ticket")Each ticket is sorted, checked (lesson 12) and scored against its label (lesson 13). Token counts feed the cost at the example prices (lesson 15), and the clock gives time per ticket (lesson 16). right += ok adds 1 for True.
Run it
python sort.pyFive of five: the login example fixed the password ticket, and nothing else broke. It also added tokens to every call, which the cost line already includes.
That is the whole loop an AI engineer runs on a prompt: change one thing, score it, look at the cost and the time, keep it or throw it away. The next step is more labelled tickets, because five cannot tell a real improvement from luck.
Where each piece came from
Choosing a model
The same script answers the question every project asks: is a small model good enough? Change name to a larger model, "Qwen/Qwen2.5-1.5B-Instruct" is three times the size, and compare the three numbers. A hosted model is the same comparison with the generate call replaced by an API call from APIs for AI. Pick the cheapest model whose score clears the bar you agreed on, not the biggest one available.
What this course left out
| Topic | What it is for |
|---|---|
| Embeddings | Turning text into vectors to find similar documents, the base of retrieval. |
| Tool calling | Letting the model ask your code to run a function, the base of agents. |
| Fine-tuning | Training a model's weights on your own examples when prompting is not enough. |
| Quantization | Storing weights in fewer bits so larger models fit in less memory. |
| Reasoning models | Models that write out intermediate steps before answering, trading tokens and time for accuracy. |
| Safety and prompt injection | Text in a ticket that tries to override your instructions; covered in the guardrails and red-teaming courses. |
You understood something today that you didn't yesterday.