The cheap model first: escalating only when needed

Most tickets are easy. Ask the cheap model first and send a ticket to the expensive one only when the cheap answer is not usable.

Example

router = Router(model_list=[
    {"model_name": "quick", "litellm_params": {"model": "shop/small"}},
    {"model_name": "careful", "litellm_params": {"model": "shop/large"}},
])


def sort_ticket(text):
    first = router.completion(model="quick", messages=[{"role": "user", "content": text}])
    answer = first.choices[0].message.content
    if answer.startswith("{"):
        return answer, "quick", first._hidden_params["response_cost"]
    second = router.completion(model="careful", messages=[{"role": "user", "content": text}])
    cost = first._hidden_params["response_cost"] + second._hidden_params["response_cost"]
    return second.choices[0].message.content, "careful", cost

Two groups from the Router lesson. sort_ticket asks quick. An answer starting with { is the JSON the app wants; anything else goes to careful, and both calls are paid for. A real app would check the answer with Pydantic, as the earlier courses did, instead of looking at the first character.

Example

tickets = ["I was charged twice for one order", "My parcel has not arrived", "How do I change my password?", "Can I get a refund?"]

total = 0
for text in tickets:
    answer, group, cost = sort_ticket(text)
    total += cost
    print(f"{group:8} ${cost:.7f}  {answer[:40]}")

all_careful = sum(router.completion(model="careful", messages=[{"role": "user", "content": t}])._hidden_params["response_cost"] for t in tickets)
print(f"escalating: ${total:.7f}   everything careful: ${all_careful:.7f}")

Three of four tickets were settled by the cheap model. The password ticket escalated, cost more than any other, and the stand-in still could not sort it, which a real careful model might. Across the batch, escalation cost about half of sending everything to the careful model, even with the most expensive ticket paying twice.

Measure before you trust it. Escalation only saves money if the cheap model's usable answers are also right. The evaluation loop from LLM Fundamentals is how to check, on labelled tickets, before switching a real workload.

Try it yourself

Make escalation happen for every ticket by lowering the bar, and compare the totals.
Count escalations in a variable and print the rate.
Change the prices so the large model is only twice the small one. Is escalation still worth it?

PreviousCaching: not paying twice for the same answer

This is what real progress feels like.