How do you design rollback for an LLM application? What exactly do you roll back?

30-second answerSay your answer out loud first, then reveal.

Roll back the specific component that changed, which requires that components are independently versioned and switchable: prompt version, model version or routing, retrieval index (keep the previous index live via aliases), guardrail config, application code, or inference server version. Make rollbacks one-click or automatic (flags, config pointers, deployment history), and test them regularly. Data changes like index rebuilds need special care, since you can only roll back if the old version still exists.

Rollback matrix

Changed component	Rollback mechanism	Prerequisite
Prompt / params	Config pointer to previous version	Prompt registry with history
Model / provider	Gateway routing config	Old model still available (watch deprecations)
RAG index	Switch alias to previous index	Keep N previous indexes during a rollback window
Embedding model	Revert alias to old index + old query embedder	Both indexes and embedders deployable
Guardrail thresholds	Config revert	Versioned guardrail configs
Code	Redeploy previous image	Standard CD history
Inference server version	Blue-green or rolling back the deployment	Previous image cached on nodes

Tricky cases

Stateful agents mid-run: in-flight tasks may have started on the new version. Record the version per run; let them finish or restart safely.
Conversation continuity: switching versions mid-conversation can confuse users. Prefer per-session stickiness.
Side effects already taken (emails sent, records updated) can't be rolled back by deployment. That requires business remediation.

PreviousExplain shadow, canary, blue-green and A/B deployments for LLM changes. When do you use each?

Slow is fine. Stopping is the only problem.

How do you design rollback for an LLM application? What exactly do you roll back?

Rollback matrix

Tricky cases

Related