Fallbacks: another model group when one fails

Ordering covers copies of one model. When the careful model itself is down, a fallback sends the request to a different group, such as a cheaper one.

router = Router(
    model_list=[
        {"model_name": "careful", "litellm_params": {"model": "down/large"}},
        {"model_name": "quick", "litellm_params": {"model": "shop/small"}},
    ],
    fallbacks=[{"careful": ["quick"]}],
    num_retries=0,
)

Two groups: careful, the expensive model the app prefers, and quick, a cheap one. fallbacks=[{"careful": ["quick"]}] says: when careful fails, try quick. The list can hold several groups, tried in order.

response = router.completion(model="careful", messages=[{"role": "user", "content": "I was charged twice for one order"}])
print(response.choices[0].message.content)
print("answered by", response.model)

The customer got an answer from the cheap model instead of an error from the careful one. Whether that is acceptable is a product decision: a worse answer now, or no answer until the provider recovers.

Two special kinds

The reliability docs name two more: context_window_fallbacks, for a request too long for one model, sent to a group with a bigger window, and content_policy_fallbacks, for a provider refusing on content grounds. Both are set the same way and only fire on their own error type.

Without a fallback

router = Router(model_list=[{"model_name": "careful", "litellm_params": {"model": "down/large"}}], num_retries=0)
try:
    router.completion(model="careful", messages=[{"role": "user", "content": "I was charged twice for one order"}])
except litellm.ServiceUnavailableError as error:
    print(str(error).splitlines()[-1])

The error message itself reports the fallbacks available for the group, here none, which is the first thing to check when an outage reaches users.