LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

Retries: trying again when a provider refuses

A retry is a second attempt at a request that failed for a reason that may pass, and a Router makes retries for you according to num_retries and retry_policy.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

A rate limit or a short server error often clears within a second. Exceptions: one set for every provider showed that LiteLLM turns every provider's errors into one set of exception classes. Retries depend on that set, because the Router looks at the kind of error before it decides to try again. The examples here use a provider that fails on demand, so they run for free and give the same output every time.

Syntax: set num_retries for every error, or retry_policy to give each kind of error its own count.

python
router = Router(
    model_list=model_list,
    num_retries=3,                        # attempts after the first one
    retry_policy=RetryPolicy(RateLimitErrorRetries=5),   # overrides it per error type
)

A provider that fails twice

The class below is a custom provider, written as in CustomLLM: writing a provider of your own. It records every call in calls and refuses the first two with a 429, the status code for a rate limit. LiteLLM turns that into RateLimitError. From the third call on it answers.

python
import litellm
from litellm import CustomLLM, Router
from litellm.llms.custom_llm import CustomLLMError   # what a custom provider raises to fail
python
calls = []                                     # one entry per call the provider receives

class Flaky(CustomLLM):
    def completion(self, model, messages, model_response, **kwargs):
        calls.append(model)
        if len(calls) <= 2:                    # the first two calls are refused
            raise CustomLLMError(status_code=429, message="rate limited")
        model_response.choices[0].message.content = "ok"
        model_response.model = model
        return model_response

litellm.custom_provider_map = [{"provider": "flaky", "custom_handler": Flaky()}]

Failing without retries

With num_retries=0 the first refusal ends the request.

ExampleNo retries
router = Router(model_list=[{"model_name": "chat", "litellm_params": {"model": "flaky/v1"}}], num_retries=0)
try:
    router.completion(model="chat", messages=[{"role": "user", "content": "hi"}])
except litellm.RateLimitError as error:
    print(type(error).__name__, "after", len(calls), "call")

Retrying until it works

With num_retries=3 the Router tries again after each refusal, up to three more attempts. The provider answers on its third call.

ExampleThree retries allowed
router = Router(model_list=[{"model_name": "chat", "litellm_params": {"model": "flaky/v1"}}], num_retries=3)
response = router.completion(model="chat", messages=[{"role": "user", "content": "hi"}])
print(response.choices[0].message.content, "after", len(calls), "calls")

The caller got an answer and never saw the two refusals. The count of calls is the only trace of them.

Which errors are retried

Retrying only helps when the next attempt can succeed. A bad request, a bad key or an unknown model will fail the same way again. The class below fails with whatever status its model name carries.

python
calls = []

class Always(CustomLLM):
    def completion(self, model, messages, model_response, **kwargs):
        calls.append(model)
        status = int(model.split("-")[1])      # the model "always/s-429" fails with a 429
        raise CustomLLMError(status_code=status, message="refused")

litellm.custom_provider_map = [{"provider": "always", "custom_handler": Always()}]
ExampleSix status codes with two retries allowed
for status in [400, 401, 404, 408, 429, 500]:
    calls.clear()
    router = Router(model_list=[{"model_name": "chat", "litellm_params": {"model": f"always/s-{status}"}}], num_retries=2)
    try:
        router.completion(model="chat", messages=[{"role": "user", "content": "hi"}])
    except Exception as error:
        print(status, type(error).__name__, "calls:", len(calls))
  • 400, 401 and 404 ran once. A malformed request, a rejected key and an unknown model do not improve with repetition, so the Router raised at once.
  • 408, 429 and 500 ran three times. A timeout, a rate limit and a server error can pass, so the Router made the first attempt and two retries.
  • The exception class still reaches your code. After the last attempt the Router raises the error that LiteLLM mapped, such as RateLimitError.

Setting a different count per error type

RetryPolicy has one field per kind of error, such as RateLimitErrorRetries and InternalServerErrorRetries, plus DefaultRetries for the rest. The policy below stops retrying rate limits and allows two retries for server errors.

python
from litellm import RetryPolicy

policy = RetryPolicy(
    RateLimitErrorRetries=0,             # a 429 is not tried again
    InternalServerErrorRetries=2,        # a 500 is tried again twice
)
router = Router(model_list=model_list, retry_policy=policy)
ExampleA policy for rate limits and server errors
from litellm import RetryPolicy

policy = RetryPolicy(RateLimitErrorRetries=0, InternalServerErrorRetries=2)
for status in [429, 500]:
    calls.clear()
    router = Router(model_list=[{"model_name": "chat", "litellm_params": {"model": f"always/s-{status}"}}], retry_policy=policy)
    try:
        router.completion(model="chat", messages=[{"role": "user", "content": "hi"}])
    except Exception as error:
        print(status, type(error).__name__, "calls:", len(calls))

The 429 ran once and the 500 ran three times, which is one attempt plus the two retries the policy allows.

num_retries vs retry_policy

num_retriesretry_policy
ScopeEvery error that can be retriedOne count per error type
Set withAn integerA RetryPolicy object
Use it forOne number for the whole RouterRate limits and server errors that need different handling
Errors that never retry400, 401 and 404 stay at one callThe same, unless you set a count for them

Where you use retries

  • Rate limits. A free-tier key that refuses a call for a second usually accepts the next one.
  • Short outages. A provider that answers 500 for a moment recovers before the second attempt.
  • Timeouts. A slow reply is attempted again before the user sees an error.
Watch out. Every retry adds waiting time, and a request that retries three times can take several times longer than one that succeeds. The Router can also retry for nothing: a group with one deployment retries the same failing provider. For failures that last, Fallbacks: another model group when one fails moves to another group. Outside a Router, completion(num_retries=3) needs the tenacity package, and 1.105.0 raised tenacity import failed when it was missing.
Try it yourself
  • Change num_retries=3 to num_retries=1 in the second example. The call fails after two attempts.
  • Add retry_after=2 to the Router in the second example and read the docs entry for what it sets.
  • In the policy example, set RateLimitErrorRetries=3 and read how many calls the 429 makes.

This is what real progress feels like.