What is model routing (or cascading), and how do you design it?

30-second answerSay your answer out loud first, then reveal.

Routing sends each request to the most suitable model: cheap and fast for easy requests, strong and expensive for hard ones. Approaches: rule-based (by feature or task type), a classifier or small LLM that predicts difficulty, or a cascade that tries the cheap model first and escalates if a confidence check or validator fails. Routing can cut cost substantially while keeping quality, but it must be evaluated, since misrouting hard queries hurts quality.

A router sends classify or extract tasks to a small model, easy queries to a mid model and hard or high-value ones to a frontier model; small and mid answers pass a validator, and failures escalate to the frontier model.

Routing strategies

Strategy	How	Pros	Cons
Static by task	Feature X always uses model Y	Simple, predictable	Doesn't adapt per query
Classifier router	Model predicts difficulty or domain	Per-query optimisation	Needs training data and eval
Cascade	Cheap first; escalate on low confidence or failed validation	Good savings	Extra latency on escalations
User tier	Premium users get the best model	Business-aligned	Quality varies per tier

Signals for escalation: schema validation failure, low log-probabilities, self-reported uncertainty, a judge model score, tool errors, user dissatisfaction (a rephrased query).

Evaluation: measure quality and cost on the routed system vs "always strong model". Track the escalation rate and misroute examples.

PreviousWhat is an LLM gateway, and why do companies build or buy one?

Slow is fine. Stopping is the only problem.

What is model routing (or cascading), and how do you design it?

Routing strategies

Related