Design online evaluation and quality monitoring for an LLM application in production.

30-second answerSay your answer out loud first, then reveal.

Log every request with full traces, then sample traffic (stratified by feature, segment and risk, plus all flagged or negative-feedback cases) for asynchronous evaluation by calibrated judges and code checks. Aggregate the results into quality metrics over time with confidence intervals, alert on regressions and anomalies, route uncertain or critical cases to human review, and feed confirmed failures back into offline datasets. Combine with explicit and implicit user signals and drift detection on inputs and outputs.

Production traces feed drift detectors, user signals and a sampler that sends traffic to async evaluators; their results land in a metrics store that drives alerts to on-call owners, while uncertain or severe cases go to a human review queue and on into offline eval datasets.

Design details

Sampling: e.g. 2% random + 100% of thumbs-down + 100% of guardrail-flagged + oversampling of rare but important segments. Weight metrics back to the population.
Evaluator costs: judges cost tokens. Budget them, use smaller judges for simple checks, and batch processing.
Latency independence: evaluation is asynchronous, so it doesn't slow user requests.
Version tagging: every metric is attributable to prompt, model and retrieval versions, for fast root-causing after deploys.
Alerting: statistical thresholds (e.g. faithfulness failure rate exceeds its 7-day baseline + 3σ), not noisy absolute triggers.
Privacy: PII scrubbing before evaluator calls if judges are external; retention limits.
Feedback loop: human-confirmed failures become regression tests.

Dashboards: quality per criterion, safety incidents, user feedback rates, latency and cost, all per feature and version.

PreviousDesign an evaluation platform for an organisation with 20 LLM-powered features built by different teams.

You understood something today that you didn't yesterday.

Design online evaluation and quality monitoring for an LLM application in production.

Design details

Related