LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

The cheap model first

The cheap model first is a routing pattern that sends every request to a small, low-cost model and calls a larger model only when the small model's answer fails a check that you wrote.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

Cooldowns: resting a failing deployment kept failing deployments out of rotation. This part is about the bill. In a support queue most tickets are easy, and a small model answers them as well as a large one for a fraction of the price. Paying the large model's price for every ticket wastes money, so the large model is held back for the tickets that need it.

A ticket goes first to the cheap model group. A check function called usable looks at the reply. If it passes, the answer is returned. If it fails, the same ticket goes to the careful group, which uses the large model, and that answer is returned instead.

Syntax: call the cheap group, test the reply, and call the careful group only if the test fails.

python
reply = router.completion(model="cheap", messages=messages, max_tokens=150)
if not usable(reply.choices[0].message.content):      # your own check
    reply = router.completion(model="careful", messages=messages, max_tokens=150)

Building the escalation

Naming a cheap group and a careful group

A Router takes a model_list. Each entry has a model_name, which is the group name your code asks for, and litellm_params, which says which real model answers. Here cheap is the 20b model and careful is the 120b model, both on Groq. The costs per token of the two models differ, and response_cost on each reply shows what a call cost.

python
import json
from litellm import Router

router = Router(model_list=[
    {"model_name": "cheap", "litellm_params": {"model": "groq/openai/gpt-oss-20b"}},
    {"model_name": "careful", "litellm_params": {"model": "groq/openai/gpt-oss-120b"}},
])

Writing the check

The check decides whether the cheap answer is good enough. It is plain Python. This one asks for JSON with a category from a fixed list, and it treats anything else as a failure. A production check would validate with Pydantic. The idea is the same.

python
ALLOWED = {"billing", "shipping", "other"}
RULE = 'Sort the ticket. Reply with JSON only, like {"category": "billing"}. Categories: billing, shipping, other.'

def usable(text):
    try:
        return json.loads(text)["category"] in ALLOWED
    except (ValueError, KeyError, TypeError):
        return False

Escalating when the check fails

sort_ticket asks cheap first and returns at once if the reply passes. If it fails, the same messages go to careful. The function adds the cost of both calls, because a failed cheap call is still paid for. response_cost can be None, so the code uses or 0.

python
def sort_ticket(ticket, **cheap_extra):
    messages = [{"role": "system", "content": RULE}, {"role": "user", "content": ticket}]
    first = router.completion(model="cheap", messages=messages, max_tokens=150, **cheap_extra)
    cost = first._hidden_params["response_cost"] or 0
    if usable(first.choices[0].message.content):
        return first.choices[0].message.content, "cheap", cost
    second = router.completion(model="careful", messages=messages, max_tokens=150)
    return second.choices[0].message.content, "careful", cost + second._hidden_params["response_cost"]

Sorting two tickets with an escalation

Two real models rarely disagree on a task this easy, so the second ticket forces the cheap call to fail. mock_response is a LiteLLM argument that makes completion() return the string you pass without calling a provider. Here it stands in for a cheap model that rambles instead of answering in JSON. The escalated call to careful is a real one.

ExampleAPI keyTwo tickets, one escalated
for ticket, extra in [
    ("I was charged twice for my order.", {}),
    ("Where is my parcel?", {"mock_response": "Hmm, hard to say."}),   # the cheap call fails the check
]:
    answer, used, cost = sort_ticket(ticket, **extra)
    print(ticket, "->", answer, "|", used, "|", f"{cost:.2e}")

Reading which model answered

  • The first line ends with cheap. The 20b model returned valid JSON, so the careful model was never called and the cost is one cheap call.
  • The second line ends with careful. The mocked reply failed usable, the same messages went to the 120b model, and its answer is the one returned.
  • The last number is the cost in dollars from response_cost. For the second ticket it includes the careful call, because that call was bought. The mocked cheap reply was never bought.
  • The wording and the cost digits change from run to run, so this block is marked live. The two routes do not.

Cheap first vs always careful

Always carefulCheap first
Calls for an easy ticketone large callone small call
Calls for a hard ticketone large callone small call and one large call
Cost when most tickets are easyhighestlow
Needs a check you trustnoyes
Delay on a hard ticketone calltwo calls

Where you use the cheap model first

  • Ticket sorting where a short label comes back and a script can check it.
  • Extraction of fields from emails, where a schema check says whether the cheap model got every field.
  • Drafting with a review step, where the small model drafts and the large model only rewrites drafts that a rule rejects.
Watch out. The check is the whole design. A check that accepts a wrong answer makes the cheap model's mistake final, and no second call happens. A check that rejects too much sends nearly every ticket to the large model and the savings disappear. Count how often each route is taken on real tickets before trusting the saving.
Try it yourself
  • Change the second ticket's mock_response to '{"category": "shipping"}'. The cheap reply now passes and the route is cheap.
  • Remove "other" from ALLOWED and send a ticket the model files under other. See the escalation fire for a good answer.
  • Print used for five tickets of your own and count how many reach careful.

You understood something today that you didn't yesterday.