After a deploy, p95 latency doubled. How do you investigate?

30-second answerSay your answer out loud first, then reveal.

Correlate first: confirm the latency jump lines up with the deploy (not provider-side or traffic changes), then compare traces before and after to see which span grew (retrieval, LLM TTFT, generation length, tool calls, retries). Typical culprits: longer prompts (more context or examples), longer outputs, extra sequential LLM calls, a different model or region, cache misses (the prompt prefix changed, so prompt caching broke), new guardrail checks, or retries from validation failures. Roll back if the SLO is burning while you investigate.

Investigation steps

Mitigate first: if the SLO error budget is burning fast, roll back (flag or config), then debug.
Diff the release bundle: what changed in code, prompt, model, retrieval or guardrails?
Per-span comparison (before vs after):

Observation	Likely cause
Input tokens +60%	Bigger prompt, more retrieved chunks, history handling bug
Output tokens +80%	Prompt change made answers verbose; max_tokens raised
TTFT up, same tokens	Prompt caching broken (dynamic content moved to the start); different model/region
Extra LLM span	New step (rewrite, judge) added sequentially
Retry spans	JSON validation failures with the new prompt
Tool spans slower	New tool or changed API usage

4. Check external factors: provider status, traffic mix shift, a canary on different infrastructure.

5. Fix and verify with a load test or canary; add a latency regression check (tokens and p95) to the CI gates.

PreviousHow do you serve many fine-tuned variants (e.g. per-customer LoRA adapters) efficiently?

Slow is fine. Stopping is the only problem.

After a deploy, p95 latency doubled. How do you investigate?

Investigation steps

Related