Cooldowns: resting a failing deployment

Retrying a provider that is down wastes time on every request. After enough failures the Router rests a deployment for a while: a cooldown.

router = Router(
    model_list=[{"model_name": "careful", "litellm_params": {"model": "down/large"}}],
    allowed_fails=1,
    cooldown_time=30,
    num_retries=0,
)

allowed_fails=1: a deployment that fails more than once in a minute is cooled down. cooldown_time=30: for 30 seconds. The routing page gives the defaults as 3 failures and 5 seconds.

for attempt in range(1, 5):
    try:
        router.completion(model="careful", messages=[{"role": "user", "content": "I was charged twice for one order"}])
    except Exception as error:
        print(attempt, type(error).__name__, "-", str(error).splitlines()[0][:60])

The first two requests reached the provider and failed with its error. After the second failure the deployment was cooled down, so the next two never reached it: the Router answered at once that no deployment was available. A request that fails fast is better than one that waits on a provider known to be down.

With a second deployment or a fallback group in place, requests during the cooldown go there instead of failing. Cooldowns apply to one deployment, not the whole group.