LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

Cooldowns: resting a failing deployment

A cooldown is a rest period during which a Router stops sending requests to a deployment that keeps failing, and tries it again when the period is over.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

Fallbacks: another model group when one fails moves a request to another group once a group has failed. A cooldown goes further. It stops later requests from reaching the failing deployment at all. Without it, every request would wait for the broken deployment to fail before moving on.

A timeline of a deployment in cooldown. The deployment bad takes traffic until its failures pass allowed_fails. It is then in cooldown and receives no traffic until cooldown_time is over, after which it is tried again. The deployment good takes traffic the whole time.

Syntax: allowed_fails sets how many failures a deployment may have in a minute, and cooldown_time sets how long it rests, in seconds. Set both yourself, because the defaults can change between versions.

python
router = Router(
    model_list=model_list,
    allowed_fails=3,        # failures in a minute before the deployment rests
    cooldown_time=30,       # seconds it rests
)

A deployment that always fails

The two classes below are custom providers, as in CustomLLM: writing a provider of your own. Bad refuses every call and records it in hits. Good always answers. The model name carries the status code Bad fails with, so the same class serves several experiments.

python
import litellm
from litellm import CustomLLM, Router
from litellm.llms.custom_llm import CustomLLMError   # what a custom provider raises to fail
import time
python
hits = []                                     # every call the failing provider receives

class Bad(CustomLLM):
    def completion(self, model, messages, model_response, **kwargs):
        hits.append(model)
        raise CustomLLMError(status_code=int(model.split("-")[1]), message="refused")

class Good(CustomLLM):
    def completion(self, model, messages, model_response, **kwargs):
        model_response.choices[0].message.content = "good"
        model_response.model = "good"
        return model_response

Registering them gives the prefixes bad/ and good/ a meaning.

python
litellm.custom_provider_map = [
    {"provider": "bad", "custom_handler": Bad()},
    {"provider": "good", "custom_handler": Good()},
]

Both go in one group. order puts Bad first, so it is tried on every call until the Router stops using it. Without order the Router would choose between them at random and the counts would change between runs.

Counting failures with allowed_fails

The loop runs five requests with allowed_fails at 0 and then at 2, and counts the calls that reached Bad. A 500 error is used because it does not cool a deployment on its first failure, as the section on error kinds shows.

ExampleFive requests, a deployment that always fails
for allowed in [0, 2]:
    hits.clear()
    router = Router(
        model_list=[
        {"model_name": "chat", "litellm_params": {"model": "bad/s-500", "order": 1}},
        {"model_name": "chat", "litellm_params": {"model": "good/v1", "order": 2}},
    ],
        allowed_fails=allowed, cooldown_time=60, num_retries=1,
    )
    for _ in range(5):
        router.completion(model="chat", messages=[{"role": "user", "content": "hi"}])
    print("allowed_fails", allowed, "-> bad called", len(hits), "times")
  • allowed_fails=0 left the failing deployment after one call. The first failure was already over the limit, so it went into cooldown and the other four requests never reached it.
  • allowed_fails=2 let it fail three times. The deployment rests once its failures pass the limit, and the three failures are what it took to pass 2.
  • All five requests were answered. The good deployment answered each one, so the cooldown cost the caller nothing.

Waiting for the cooldown to end

A cooldown is temporary. The next example sets cooldown_time=2, makes a request, repeats it at once, and then waits three seconds before a third.

ExampleA two second cooldown
router = Router(
    model_list=[
        {"model_name": "chat", "litellm_params": {"model": "bad/s-500", "order": 1}},
        {"model_name": "chat", "litellm_params": {"model": "good/v1", "order": 2}},
    ],
    allowed_fails=0, cooldown_time=2, num_retries=1,
)

def ask(label):
    response = router.completion(model="chat", messages=[{"role": "user", "content": "hi"}])
    print(label, response.choices[0].message.content, "| bad called", len(hits))

ask("first request ")
ask("at once       ")
time.sleep(3)
ask("after 3 s     ")
  • first request failed once on Bad, then Good answered. The count is 1.
  • at once skipped Bad, because it was resting. The count stayed 1.
  • after 3 s tried Bad again, because the cooldown was over. It failed again, so the count is 2 and the deployment rested again.

Which errors cool a deployment down

A cooldown does not always wait for allowed_fails. The next example leaves allowed_fails unset and sends four requests for each kind of failure. It prints how many calls had reached Bad after each request, so you can see when the calls stop.

ExampleFour requests per kind of failure, counted after each one
for status in [429, 401, 404, 500]:
    hits.clear()
    router = Router(model_list=[
        {"model_name": "chat", "litellm_params": {"model": f"bad/s-{status}", "order": 1}},
        {"model_name": "chat", "litellm_params": {"model": "good/v1", "order": 2}},
    ], num_retries=3, cooldown_time=60)
    after_each = []
    for _ in range(4):
        router.completion(model="chat", messages=[{"role": "user", "content": "hi"}])
        after_each.append(len(hits))
    print(status, "-> calls that reached bad after each request:", after_each)
  • 429, 401 and 404 cooled the deployment after one failure. The count is 1 and never grows. A rate limit, a rejected key and an unknown model will not fix themselves, so the Router rests the deployment at once. The library's own comment lists these three. A 429 does this only when the group has another deployment to use.
  • 500 was called four times by the first request. It tried the failing deployment once and retried it three times, because order keeps it first. The second request added a fifth failure, and then the count stopped at 5.
  • Why 500 stopped at five. With allowed_fails unset, the library also rests a deployment when more than half of its calls in the last minute failed and at least five calls were made. Here all five had failed. A server error that comes and goes does not trip this rule. A steady one does. Setting allowed_fails yourself replaces this rule with a plain count.

Cooldown vs fallback

CooldownFallback
Acts onA deploymentA model group
RemembersYes, for cooldown_time secondsNo, each request decides again
SavesThe wait for a failing deployment on later requestsThe request that failed
Set withallowed_fails and cooldown_timefallbacks

Where you use cooldowns

  • A key that hits its rate limit. The deployment rests while the limit resets, and the group's other keys carry the traffic.
  • A provider that is down. Requests stop paying the delay of a failing call before reaching a working one.
  • A key that has been revoked. A 401 takes the deployment out of rotation until someone fixes it.
Watch out. A cooldown only helps when the group has another deployment to use. If every deployment is resting, or the group has only one, requests have nowhere to go and fail. Keep at least two deployments in any group that must stay up, and set cooldown_time short enough that a recovered provider is used again quickly.
Try it yourself
  • Change cooldown_time=2 to cooldown_time=10 in the waiting example. The request after 3 s no longer reaches Bad.
  • Set allowed_fails=1 in the counting example and read how many calls reach Bad.
  • Swap order so that Good is first, and confirm that Bad is never called.

Little by little, you're building something great.