Write the summary of a post-mortem for an LLM incident: an agent's retry loop caused a ₹4 lakh overnight cost spike.
Example post-mortem (condensed)
Summary: Between 01:10 and 07:45 IST, the document-processing agent repeatedly retried a failing extraction step for ~3,200 documents, generating ~210M extra tokens and ₹4.1 lakh in unplanned cost. No customer data was exposed; processing of the affected documents was delayed by 9 hours.
Impact: ₹4.1 lakh cost; 3,200 documents delayed; no data or security impact.
Timeline (IST):
| Time | Event |
|---|---|
| 00:55 | Release 2026.10.05-2 deployed (new extraction prompt with stricter JSON schema) |
| 01:10 | Schema validation failures begin on scanned documents; agent retries without limit |
| 02:00 | Cost rises to 6x normal hourly rate (no alert configured for hourly anomalies) |
| 07:30 | Engineer notices the daily cost dashboard; incident declared |
| 07:45 | Feature flag rolls back the prompt; retries stop |
Root cause: the new schema required a field that scanned documents often lack. The agent's validation-retry path had no maximum attempt count per document, and the retry prompt didn't allow "field not found".
Contributing factors: the eval set had few scanned documents; no per-run token budget; cost alerts were only daily; staging used a smaller sample.
What went well: the flag-based rollback took 2 minutes; traces made the root cause clear quickly.
Action items:
| Action | Owner | Due |
|---|---|---|
| Max 3 validation retries per document; then route to human review | Agent team | Oct 10 |
| Per-run and per-hour token budgets with hard stops | Platform | Oct 14 |
| Hourly cost anomaly alerts (page above 3x baseline) | Platform | Oct 9 |
| Add 50 scanned documents to the golden set; include "missing field" cases | Agent team | Oct 12 |
| Canary cost check: tokens per task within +20% before expanding | Platform | Oct 16 |
Related
You understood something today that you didn't yesterday.