How do you A/B test LLM features in production?

30-second answerSay your answer out loud first, then reveal.

Randomise at the right unit (user or account, not request, to avoid inconsistent experiences), compare the variant (new prompt, model or retrieval) against control on a primary business metric with pre-registered guardrail metrics (quality, safety, latency, cost). Size the experiment for statistical power, run it long enough to cover weekly cycles and novelty effects, and combine with offline evals and sampled human or LLM-judge review, because online metrics are noisy and lagging.

Steps

Offline gate first: only variants that pass offline evals go to A/B.
Hypothesis + metrics: e.g. "new retrieval increases resolution rate by 3% without raising escalations or cost per conversation > 10%."
Randomisation unit: user/account level; sticky assignment.
Sample size / power: compute from baseline rate and minimum detectable effect. LLM features often have high variance, so plan for enough traffic.
Guardrails: latency p95, error rate, safety flags, cost. Auto-stop if breached.
Duration: at least 1–2 weeks for weekly seasonality; watch novelty effects.
Analysis: segment by user type, language and query type, since averages hide regressions.

LLM-specific twists

Online quality measurement: sample conversations from both arms for blind LLM-judge or human grading.
Interleaving for ranking or search changes (more sensitive than A/B).
Cost as a first-class metric: a variant can win on quality but lose on unit economics.
Model provider changes can affect both arms. Pin versions during the experiment.

Common mistakes

Per-request randomisation in chat (the user sees different personalities between turns).
Declaring victory from a 2-day test on thumbs-up rate.

PreviousDesign AI-powered semantic product search for an e-commerce site with 10M products.

Slow is fine. Stopping is the only problem.

How do you A/B test LLM features in production?

Steps

LLM-specific twists

Common mistakes

Related