Explain shadow, canary, blue-green and A/B deployments for LLM changes. When do you use each?

30-second answerSay your answer out loud first, then reveal.

Shadow runs the new version on real traffic without showing users its output, which is safe for comparing quality, latency and cost. Canary serves the new version to a small percentage of traffic, with automated metric comparison before expanding. Blue-green keeps two full environments and switches traffic at once, with instant rollback. A/B tests split users randomly over a longer period to measure the impact on business outcomes. LLM changes often go shadow → canary → full, with A/B tests when the change targets user outcomes.

A router sends 100% of live traffic to the current version, a copy to a shadow version whose outputs are logged and judged but not served, and 5% to a canary; a comparison of errors, latency, cost, judge scores and feedback then either expands the rollout to 25%, 50% and 100% or rolls back.

When to use each

Strategy	User exposure	Best for	Cost / caveat
Shadow	None	Model swaps, big prompt rewrites, risky retrieval changes	Doubles inference cost for shadowed traffic; can't measure user reactions; avoid side-effecting tools in shadow
Canary	Small %	Most releases	Needs enough traffic for signal; automated analysis
Blue-green	All at once (switchable)	Infrastructure changes, inference-server upgrades	Duplicate infrastructure (expensive for GPUs)
A/B test	Split for weeks	Measuring business impact of prompt/model changes	Requires experiment design and statistical rigour

LLM-specific note: shadowing agents is tricky because tools have side effects. Run shadows with read-only or mocked write tools.

PreviousDesign a CI/CD pipeline for an LLM application with evaluation gates.

You understood something today that you didn't yesterday.

Explain shadow, canary, blue-green and A/B deployments for LLM changes. When do you use each?

When to use each

Related