How do you handle provider rate limits and quotas in an LLM system?

30-second answerSay your answer out loud first, then reveal.

Know the limits (requests per minute, tokens per minute, concurrent requests) per model and key. Enforce your own client-side rate limiting and queueing so you shape traffic instead of hitting 429s. Retry with exponential backoff and jitter, respecting retry-after headers. Spread load across keys, regions or providers where permitted. Prioritise interactive over batch traffic, and request quota increases or reserved capacity ahead of known peaks.

Techniques

Token-aware rate limiter: estimate tokens per request (input + max output) and use a token bucket per model, so you count tokens rather than just requests.
Queues with priorities: interactive requests first; batch jobs fill spare capacity.
Backoff: exponential with jitter on 429/529/503; cap total retry time (users won't wait 2 minutes).
Load shedding / graceful degradation: under pressure, route to a smaller model, shorten context, disable non-essential AI features, or show a "high demand" message.
Multi-provider / multi-region failover via the gateway (Q7).
Capacity planning: provisioned throughput or reserved capacity offerings; quota increase requests before launches and sales.
Reduce demand: caching, smaller prompts, batching of offline work.

Monitoring: 429 rate, queue wait time, tokens per minute used vs limit.

Common mistakes

Retrying immediately in tight loops (thundering herd), which makes the limit problem worse.

PreviousWhat is model routing (or cascading), and how do you design it?

This is what real progress feels like.

How do you handle provider rate limits and quotas in an LLM system?

Techniques

Common mistakes

Related