Rate limits: 429 and Retry-After

Every model API limits how many requests you may send in a period. Past the limit it answers 429, and a program that sends thousands of tickets has to wait and try again.

Give the stand-in a limit, so the behaviour can be seen. It refuses every third request:

Examplemodel_api.py, new

calls = {"count": 0}


def check_rate_limit():
    calls["count"] += 1
    if calls["count"] % 3 == 0:
        raise HTTPException(status_code=429, detail="rate limit reached", headers={"Retry-After": "1"})

check_rate_limit() is called in the endpoint straight after check_key. The Retry-After header says how many seconds to wait. Real APIs count requests and tokens per minute instead of every third call, but they answer the same way.

Example

from fastapi.testclient import TestClient
from model_api import app

model = TestClient(app, headers={"Authorization": "Bearer sk-local-123"})
body = {"model": "local", "messages": [{"role": "user", "content": "My parcel has not arrived"}]}

for _ in range(4):
    response = model.post("/v1/chat/completions", json=body)
    print(response.status_code, response.headers.get("retry-after"))

Waiting and trying again

Example

def ask(model, text, attempts=3):
    body = {"model": "local", "messages": [{"role": "user", "content": text}]}
    for attempt in range(1, attempts + 1):
        response = model.post("/v1/chat/completions", json=body)
        if response.status_code != 429:
            response.raise_for_status()
            return response.json()["choices"][0]["message"]["content"]
        wait = int(response.headers.get("retry-after", "1"))
        print(f"attempt {attempt}: 429, waiting {wait}s")
        time.sleep(wait)
    raise RuntimeError(f"still rate limited after {attempts} attempts")

for text in ["I was charged twice", "My parcel has not arrived", "Refund please"]:
    print(ask(model, text))

A 429 is the one error worth retrying unchanged, so ask checks for it first. Any other error still goes to raise_for_status, because sending a 401 again will only get another 401.

It waits as long as the API asked, not a number of its own. When the header is missing, one second is a reasonable guess. After three refusals it stops with an error instead of looping forever.

The third ticket hit the limit, waited a second and went through. The retries lesson in Python for AI added backoff, waiting longer each time; when an API sends Retry-After, use that instead.

Try it yourself

Change the limit to % 2 and run the loop.
Set attempts=1 and find the ticket that fails.
Remove the Retry-After header from the server and check the default is used.

PreviousAPI keys: headers and environment variables

Little by little, you're building something great.