Cooldowns: resting a failing deployment
A cooldown is a rest period during which a Router stops sending requests to a deployment that keeps failing, and tries it again when the period is over.
Last updated: 12 Oct, 2026 · LiteLLM 1.105
Fallbacks: another model group when one fails moves a request to another group once a group has failed. A cooldown goes further. It stops later requests from reaching the failing deployment at all. Without it, every request would wait for the broken deployment to fail before moving on.
Syntax: allowed_fails sets how many failures a deployment may have in a minute, and cooldown_time sets how long it rests, in seconds. Set both yourself, because the defaults can change between versions.
router = Router(
model_list=model_list,
allowed_fails=3, # failures in a minute before the deployment rests
cooldown_time=30, # seconds it rests
)A deployment that always fails
The two classes below are custom providers, as in CustomLLM: writing a provider of your own. Bad refuses every call and records it in hits. Good always answers. The model name carries the status code Bad fails with, so the same class serves several experiments.
import litellm
from litellm import CustomLLM, Router
from litellm.llms.custom_llm import CustomLLMError # what a custom provider raises to fail
import timehits = [] # every call the failing provider receives
class Bad(CustomLLM):
def completion(self, model, messages, model_response, **kwargs):
hits.append(model)
raise CustomLLMError(status_code=int(model.split("-")[1]), message="refused")
class Good(CustomLLM):
def completion(self, model, messages, model_response, **kwargs):
model_response.choices[0].message.content = "good"
model_response.model = "good"
return model_responseRegistering them gives the prefixes bad/ and good/ a meaning.
litellm.custom_provider_map = [
{"provider": "bad", "custom_handler": Bad()},
{"provider": "good", "custom_handler": Good()},
]Both go in one group. order puts Bad first, so it is tried on every call until the Router stops using it. Without order the Router would choose between them at random and the counts would change between runs.
Counting failures with allowed_fails
The loop runs five requests with allowed_fails at 0 and then at 2, and counts the calls that reached Bad. A 500 error is used because it does not cool a deployment on its first failure, as the section on error kinds shows.
for allowed in [0, 2]:
hits.clear()
router = Router(
model_list=[
{"model_name": "chat", "litellm_params": {"model": "bad/s-500", "order": 1}},
{"model_name": "chat", "litellm_params": {"model": "good/v1", "order": 2}},
],
allowed_fails=allowed, cooldown_time=60, num_retries=1,
)
for _ in range(5):
router.completion(model="chat", messages=[{"role": "user", "content": "hi"}])
print("allowed_fails", allowed, "-> bad called", len(hits), "times")allowed_fails 0 -> bad called 1 times allowed_fails 2 -> bad called 3 times
- allowed_fails=0 left the failing deployment after one call. The first failure was already over the limit, so it went into cooldown and the other four requests never reached it.
- allowed_fails=2 let it fail three times. The deployment rests once its failures pass the limit, and the three failures are what it took to pass 2.
- All five requests were answered. The good deployment answered each one, so the cooldown cost the caller nothing.
Waiting for the cooldown to end
A cooldown is temporary. The next example sets cooldown_time=2, makes a request, repeats it at once, and then waits three seconds before a third.
router = Router(
model_list=[
{"model_name": "chat", "litellm_params": {"model": "bad/s-500", "order": 1}},
{"model_name": "chat", "litellm_params": {"model": "good/v1", "order": 2}},
],
allowed_fails=0, cooldown_time=2, num_retries=1,
)
def ask(label):
response = router.completion(model="chat", messages=[{"role": "user", "content": "hi"}])
print(label, response.choices[0].message.content, "| bad called", len(hits))
ask("first request ")
ask("at once ")
time.sleep(3)
ask("after 3 s ")first request good | bad called 1 at once good | bad called 1 after 3 s good | bad called 2
- first request failed once on
Bad, thenGoodanswered. The count is 1. - at once skipped
Bad, because it was resting. The count stayed 1. - after 3 s tried
Badagain, because the cooldown was over. It failed again, so the count is 2 and the deployment rested again.
Which errors cool a deployment down
A cooldown does not always wait for allowed_fails. The next example leaves allowed_fails unset and sends four requests for each kind of failure. It prints how many calls had reached Bad after each request, so you can see when the calls stop.
for status in [429, 401, 404, 500]:
hits.clear()
router = Router(model_list=[
{"model_name": "chat", "litellm_params": {"model": f"bad/s-{status}", "order": 1}},
{"model_name": "chat", "litellm_params": {"model": "good/v1", "order": 2}},
], num_retries=3, cooldown_time=60)
after_each = []
for _ in range(4):
router.completion(model="chat", messages=[{"role": "user", "content": "hi"}])
after_each.append(len(hits))
print(status, "-> calls that reached bad after each request:", after_each)429 -> calls that reached bad after each request: [1, 1, 1, 1] 401 -> calls that reached bad after each request: [1, 1, 1, 1] 404 -> calls that reached bad after each request: [1, 1, 1, 1] 500 -> calls that reached bad after each request: [4, 5, 5, 5]
- 429, 401 and 404 cooled the deployment after one failure. The count is 1 and never grows. A rate limit, a rejected key and an unknown model will not fix themselves, so the Router rests the deployment at once. The library's own comment lists these three. A 429 does this only when the group has another deployment to use.
- 500 was called four times by the first request. It tried the failing deployment once and retried it three times, because
orderkeeps it first. The second request added a fifth failure, and then the count stopped at 5. - Why 500 stopped at five. With
allowed_failsunset, the library also rests a deployment when more than half of its calls in the last minute failed and at least five calls were made. Here all five had failed. A server error that comes and goes does not trip this rule. A steady one does. Settingallowed_failsyourself replaces this rule with a plain count.
Cooldown vs fallback
| Cooldown | Fallback | |
|---|---|---|
| Acts on | A deployment | A model group |
| Remembers | Yes, for cooldown_time seconds | No, each request decides again |
| Saves | The wait for a failing deployment on later requests | The request that failed |
| Set with | allowed_fails and cooldown_time | fallbacks |
Where you use cooldowns
- A key that hits its rate limit. The deployment rests while the limit resets, and the group's other keys carry the traffic.
- A provider that is down. Requests stop paying the delay of a failing call before reaching a working one.
- A key that has been revoked. A 401 takes the deployment out of rotation until someone fixes it.
cooldown_time short enough that a recovered provider is used again quickly.Related
- Previous: Fallbacks: another model group when one fails
- Next: The cheap model first
- See also: Routing strategies: choosing which deployment answers for the
ordersetting used in these examples - Reference: Cooldowns in the LiteLLM docs
- Change
cooldown_time=2tocooldown_time=10in the waiting example. The request after 3 s no longer reachesBad. - Set
allowed_fails=1in the counting example and read how many calls reachBad. - Swap
orderso thatGoodis first, and confirm thatBadis never called.
Little by little, you're building something great.