Users see intermittent timeouts because the LLM provider's tail latency is high. How do you make the system robust?

30-second answerSay your answer out loud first, then reveal.

Set sensible timeouts per call based on the expected output length (and a TTFT timeout for streaming), retry with backoff on retryable errors only, use hedged requests for latency-critical paths (send a second request after a delay to another region, provider or model, and use whichever responds first), route around degraded endpoints with health-based routing and circuit breakers, reduce tail sensitivity (shorter outputs, smaller models for simple steps), and design the UX for slow responses (streaming, progress, async completion).

Techniques

Timeouts: TTFT timeout (e.g. 5s) + total timeout scaled to max_tokens; don't use one fixed 30s for everything.
Retries: only for timeouts, 429s and 5xx; exponential backoff + jitter; cap the attempts and the total time budget.
Hedging: for short, idempotent calls (classification, routing), fire a backup request after the p95 latency elapses; cancel the loser. It increases cost slightly and cuts tail latency significantly.
Health-aware routing: track latency and error rates per region and provider; shift traffic away from degraded endpoints automatically.
Circuit breakers: stop hammering a failing endpoint; fail over or degrade.
Graceful degradation: a smaller or faster model, cached answers, "We're taking longer than usual" messaging, async completion with notification.
Observability: p99 latency per provider, region and model; timeout and retry rates; alerting.

Caution: retries and hedges multiply load during incidents. Use retry budgets to avoid amplifying an outage.

PreviousYour self-hosted vLLM servers start throwing out-of-memory errors or preempting requests under load. What do you do?

Every expert started right here.

Users see intermittent timeouts because the LLM provider's tail latency is high. How do you make the system robust?

Techniques

Related