Retries: trying again when a provider refuses

A rate limit usually clears in seconds. num_retries tells LiteLLM to try again before giving up, so a brief refusal never reaches your users.

Example

import litellm
from litellm import completion

import flaky_llm

litellm.suppress_debug_info = True

response = completion(model="flaky/small", messages=[{"role": "user", "content": "I was charged twice for one order"}], num_retries=2)
print(response.choices[0].message.content)
print("calls made:", flaky_llm.flaky.calls)

flaky/small refuses its first two calls. With num_retries=2 LiteLLM made the first call and two retries, and the third call answered. The caller saw only the answer.

Not enough retries

Example

try:
    completion(model="flaky/small", messages=[{"role": "user", "content": "I was charged twice for one order"}], num_retries=1)
except litellm.RateLimitError:
    print("still rate limited after", flaky_llm.flaky.calls, "calls")

One retry is two calls, both refused, so the error reaches you. Set retries for how long an outage you are willing to wait through, not as high as possible: each retry adds time to a request someone is waiting on.

Retries need tenacity

In this version, num_retries on completion() uses the tenacity package and fails with tenacity import failed please run `pip install tenacity` when it is missing, which is why lesson 0 installs it. The Router in the next lesson retries without it.

Try it yourself

Make FlakyLLM(failures=5) and find the smallest num_retries that works.
Retry down/large three times. Does it ever answer?
Time a call with 2 retries using time.perf_counter.

PreviousCallbacks: tracking every call

This is what real progress feels like.