The cheap model first
The cheap model first is a routing pattern that sends every request to a small, low-cost model and calls a larger model only when the small model's answer fails a check that you wrote.
Last updated: 12 Oct, 2026 · LiteLLM 1.105
Cooldowns: resting a failing deployment kept failing deployments out of rotation. This part is about the bill. In a support queue most tickets are easy, and a small model answers them as well as a large one for a fraction of the price. Paying the large model's price for every ticket wastes money, so the large model is held back for the tickets that need it.
Syntax: call the cheap group, test the reply, and call the careful group only if the test fails.
reply = router.completion(model="cheap", messages=messages, max_tokens=150)
if not usable(reply.choices[0].message.content): # your own check
reply = router.completion(model="careful", messages=messages, max_tokens=150)Building the escalation
Naming a cheap group and a careful group
A Router takes a model_list. Each entry has a model_name, which is the group name your code asks for, and litellm_params, which says which real model answers. Here cheap is the 20b model and careful is the 120b model, both on Groq. The costs per token of the two models differ, and response_cost on each reply shows what a call cost.
import json
from litellm import Router
router = Router(model_list=[
{"model_name": "cheap", "litellm_params": {"model": "groq/openai/gpt-oss-20b"}},
{"model_name": "careful", "litellm_params": {"model": "groq/openai/gpt-oss-120b"}},
])Writing the check
The check decides whether the cheap answer is good enough. It is plain Python. This one asks for JSON with a category from a fixed list, and it treats anything else as a failure. A production check would validate with Pydantic. The idea is the same.
ALLOWED = {"billing", "shipping", "other"}
RULE = 'Sort the ticket. Reply with JSON only, like {"category": "billing"}. Categories: billing, shipping, other.'
def usable(text):
try:
return json.loads(text)["category"] in ALLOWED
except (ValueError, KeyError, TypeError):
return FalseEscalating when the check fails
sort_ticket asks cheap first and returns at once if the reply passes. If it fails, the same messages go to careful. The function adds the cost of both calls, because a failed cheap call is still paid for. response_cost can be None, so the code uses or 0.
def sort_ticket(ticket, **cheap_extra):
messages = [{"role": "system", "content": RULE}, {"role": "user", "content": ticket}]
first = router.completion(model="cheap", messages=messages, max_tokens=150, **cheap_extra)
cost = first._hidden_params["response_cost"] or 0
if usable(first.choices[0].message.content):
return first.choices[0].message.content, "cheap", cost
second = router.completion(model="careful", messages=messages, max_tokens=150)
return second.choices[0].message.content, "careful", cost + second._hidden_params["response_cost"]Sorting two tickets with an escalation
Two real models rarely disagree on a task this easy, so the second ticket forces the cheap call to fail. mock_response is a LiteLLM argument that makes completion() return the string you pass without calling a provider. Here it stands in for a cheap model that rambles instead of answering in JSON. The escalated call to careful is a real one.
for ticket, extra in [
("I was charged twice for my order.", {}),
("Where is my parcel?", {"mock_response": "Hmm, hard to say."}), # the cheap call fails the check
]:
answer, used, cost = sort_ticket(ticket, **extra)
print(ticket, "->", answer, "|", used, "|", f"{cost:.2e}")I was charged twice for my order. -> {"category":"billing"} | cheap | 3.47e-05
Where is my parcel? -> {"category": "shipping"} | careful | 4.64e-05Reading which model answered
- The first line ends with
cheap. The 20b model returned valid JSON, so the careful model was never called and the cost is one cheap call. - The second line ends with
careful. The mocked reply failedusable, the same messages went to the 120b model, and its answer is the one returned. - The last number is the cost in dollars from
response_cost. For the second ticket it includes the careful call, because that call was bought. The mocked cheap reply was never bought. - The wording and the cost digits change from run to run, so this block is marked live. The two routes do not.
Cheap first vs always careful
| Always careful | Cheap first | |
|---|---|---|
| Calls for an easy ticket | one large call | one small call |
| Calls for a hard ticket | one large call | one small call and one large call |
| Cost when most tickets are easy | highest | low |
| Needs a check you trust | no | yes |
| Delay on a hard ticket | one call | two calls |
Where you use the cheap model first
- Ticket sorting where a short label comes back and a script can check it.
- Extraction of fields from emails, where a schema check says whether the cheap model got every field.
- Drafting with a review step, where the small model drafts and the large model only rewrites drafts that a rule rejects.
Related
- Previous: Cooldowns: resting a failing deployment
- Next: Caching responses
- See also: Router: model groups and deployments, Tokens and cost
- Reference: Mock completion responses in the LiteLLM docs
- Change the second ticket's
mock_responseto'{"category": "shipping"}'. The cheap reply now passes and the route ischeap. - Remove
"other"fromALLOWEDand send a ticket the model files under other. See the escalation fire for a good answer. - Print
usedfor five tickets of your own and count how many reachcareful.
You understood something today that you didn't yesterday.