What is model routing (or cascading), and how do you design it?
30-second answerSay your answer out loud first, then reveal.
Routing strategies
| Strategy | How | Pros | Cons |
|---|---|---|---|
| Static by task | Feature X always uses model Y | Simple, predictable | Doesn't adapt per query |
| Classifier router | Model predicts difficulty or domain | Per-query optimisation | Needs training data and eval |
| Cascade | Cheap first; escalate on low confidence or failed validation | Good savings | Extra latency on escalations |
| User tier | Premium users get the best model | Business-aligned | Quality varies per tier |
Signals for escalation: schema validation failure, low log-probabilities, self-reported uncertainty, a judge model score, tool errors, user dissatisfaction (a rephrased query).
Evaluation: measure quality and cost on the routed system vs "always strong model". Track the escalation rate and misroute examples.
Related
Slow is fine. Stopping is the only problem.