1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
51 small wins to finish your pathNext question →
You need to switch the agent's model to a newer or cheaper one. After switching, some behaviours regress. How do you manage model migrations?
30-second answerSay your answer out loud first, then reveal.
Process
- Pin versions: use dated model identifiers; a silent provider update shouldn't change behaviour.
- Eval suite comparison:
• Task success, pass^k, tool-selection accuracy.
• Safety/policy tests and injection resistance.
• Cost per successful task, p50/p95 latency, steps per task.
• Output format adherence (JSON, length, tone). - Diff analysis: run the same tasks on both and compare trajectories. New models often differ in:
• How literally they follow instructions (newer models can be more literal, so old workarounds and "SHOUTING" prompts can now over-trigger).
• How eagerly they call tools, or call them in parallel.
• Verbosity and refusal behaviour. - Re-tune: prompts are partially model-specific. Remove old hacks and adjust examples and tool descriptions. Re-check your LLM judges if the judge model changed too.
- Rollout: shadow mode (new model runs, old model serves) → compare → canary 5% → ramp by segment, with automatic rollback triggers on key metrics.
- Keep a fallback model configured for outages.
Mention abstraction: a model gateway or router layer (LiteLLM, a cloud gateway, an internal service) makes swapping, A/B testing and fallbacks configuration rather than code changes.
Related
Slow is fine. Stopping is the only problem.