Retries: trying again when a provider refuses
A retry is a second attempt at a request that failed for a reason that may pass, and a Router makes retries for you according to num_retries and retry_policy.
Last updated: 12 Oct, 2026 · LiteLLM 1.105
A rate limit or a short server error often clears within a second. Exceptions: one set for every provider showed that LiteLLM turns every provider's errors into one set of exception classes. Retries depend on that set, because the Router looks at the kind of error before it decides to try again. The examples here use a provider that fails on demand, so they run for free and give the same output every time.
Syntax: set num_retries for every error, or retry_policy to give each kind of error its own count.
router = Router(
model_list=model_list,
num_retries=3, # attempts after the first one
retry_policy=RetryPolicy(RateLimitErrorRetries=5), # overrides it per error type
)A provider that fails twice
The class below is a custom provider, written as in CustomLLM: writing a provider of your own. It records every call in calls and refuses the first two with a 429, the status code for a rate limit. LiteLLM turns that into RateLimitError. From the third call on it answers.
import litellm
from litellm import CustomLLM, Router
from litellm.llms.custom_llm import CustomLLMError # what a custom provider raises to failcalls = [] # one entry per call the provider receives
class Flaky(CustomLLM):
def completion(self, model, messages, model_response, **kwargs):
calls.append(model)
if len(calls) <= 2: # the first two calls are refused
raise CustomLLMError(status_code=429, message="rate limited")
model_response.choices[0].message.content = "ok"
model_response.model = model
return model_response
litellm.custom_provider_map = [{"provider": "flaky", "custom_handler": Flaky()}]Failing without retries
With num_retries=0 the first refusal ends the request.
router = Router(model_list=[{"model_name": "chat", "litellm_params": {"model": "flaky/v1"}}], num_retries=0)
try:
router.completion(model="chat", messages=[{"role": "user", "content": "hi"}])
except litellm.RateLimitError as error:
print(type(error).__name__, "after", len(calls), "call")RateLimitError after 1 call
Retrying until it works
With num_retries=3 the Router tries again after each refusal, up to three more attempts. The provider answers on its third call.
router = Router(model_list=[{"model_name": "chat", "litellm_params": {"model": "flaky/v1"}}], num_retries=3)
response = router.completion(model="chat", messages=[{"role": "user", "content": "hi"}])
print(response.choices[0].message.content, "after", len(calls), "calls")ok after 3 calls
The caller got an answer and never saw the two refusals. The count of calls is the only trace of them.
Which errors are retried
Retrying only helps when the next attempt can succeed. A bad request, a bad key or an unknown model will fail the same way again. The class below fails with whatever status its model name carries.
calls = []
class Always(CustomLLM):
def completion(self, model, messages, model_response, **kwargs):
calls.append(model)
status = int(model.split("-")[1]) # the model "always/s-429" fails with a 429
raise CustomLLMError(status_code=status, message="refused")
litellm.custom_provider_map = [{"provider": "always", "custom_handler": Always()}]for status in [400, 401, 404, 408, 429, 500]:
calls.clear()
router = Router(model_list=[{"model_name": "chat", "litellm_params": {"model": f"always/s-{status}"}}], num_retries=2)
try:
router.completion(model="chat", messages=[{"role": "user", "content": "hi"}])
except Exception as error:
print(status, type(error).__name__, "calls:", len(calls))400 BadRequestError calls: 1 401 AuthenticationError calls: 1 404 NotFoundError calls: 1 408 Timeout calls: 3 429 RateLimitError calls: 3 500 InternalServerError calls: 3
- 400, 401 and 404 ran once. A malformed request, a rejected key and an unknown model do not improve with repetition, so the Router raised at once.
- 408, 429 and 500 ran three times. A timeout, a rate limit and a server error can pass, so the Router made the first attempt and two retries.
- The exception class still reaches your code. After the last attempt the Router raises the error that LiteLLM mapped, such as
RateLimitError.
Setting a different count per error type
RetryPolicy has one field per kind of error, such as RateLimitErrorRetries and InternalServerErrorRetries, plus DefaultRetries for the rest. The policy below stops retrying rate limits and allows two retries for server errors.
from litellm import RetryPolicy
policy = RetryPolicy(
RateLimitErrorRetries=0, # a 429 is not tried again
InternalServerErrorRetries=2, # a 500 is tried again twice
)
router = Router(model_list=model_list, retry_policy=policy)from litellm import RetryPolicy
policy = RetryPolicy(RateLimitErrorRetries=0, InternalServerErrorRetries=2)
for status in [429, 500]:
calls.clear()
router = Router(model_list=[{"model_name": "chat", "litellm_params": {"model": f"always/s-{status}"}}], retry_policy=policy)
try:
router.completion(model="chat", messages=[{"role": "user", "content": "hi"}])
except Exception as error:
print(status, type(error).__name__, "calls:", len(calls))429 RateLimitError calls: 1 500 InternalServerError calls: 3
The 429 ran once and the 500 ran three times, which is one attempt plus the two retries the policy allows.
num_retries vs retry_policy
num_retries | retry_policy | |
|---|---|---|
| Scope | Every error that can be retried | One count per error type |
| Set with | An integer | A RetryPolicy object |
| Use it for | One number for the whole Router | Rate limits and server errors that need different handling |
| Errors that never retry | 400, 401 and 404 stay at one call | The same, unless you set a count for them |
Where you use retries
- Rate limits. A free-tier key that refuses a call for a second usually accepts the next one.
- Short outages. A provider that answers 500 for a moment recovers before the second attempt.
- Timeouts. A slow reply is attempted again before the user sees an error.
completion(num_retries=3) needs the tenacity package, and 1.105.0 raised tenacity import failed when it was missing.Related
- Previous: Routing strategies: choosing which deployment answers
- Next: Fallbacks: another model group when one fails
- See also: Exceptions: one set for every provider for the exception classes the retry rules read
- Reference: Retries in the LiteLLM docs
- Change
num_retries=3tonum_retries=1in the second example. The call fails after two attempts. - Add
retry_after=2to the Router in the second example and read the docs entry for what it sets. - In the policy example, set
RateLimitErrorRetries=3and read how many calls the 429 makes.
This is what real progress feels like.