LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

Fallbacks: another model group when one fails

A fallback is a backup model group that a Router tries when the group you asked for fails, so one provider's outage does not become your outage.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

Retries: trying again when a provider refuses repeated a request against the same group. A fallback moves the request to a different group. It is the next step when repeating does not help, such as a model that has been retired or a provider that is down for an hour.

A fallback chain. A call to the group primary fails, so the Router tries the group second, which also fails, and then the group third, which answers. The fallbacks setting lists the groups in order. Each group is retried num_retries times before the Router moves to the next group.

Syntax: fallbacks is a list of dictionaries. Each key is a group name and each value is the list of groups to try, in order, when that group fails.

python
router = Router(
    model_list=model_list,
    fallbacks=[{"primary": ["backup"]}],    # when "primary" fails, try "backup"
    num_retries=0,
)

Seeing the failure without a fallback

The cleanest way to prove a fallback fired is a primary model that cannot answer. The name groq/this-model-does-not-exist is not a model, so every call to it fails.

ExampleA primary that does not exist, and no fallback
import litellm
from litellm import Router

litellm.suppress_debug_info = True     # hides LiteLLM's help banner on errors

router = Router(
    model_list=[{"model_name": "primary", "litellm_params": {"model": "groq/this-model-does-not-exist"}}],
    num_retries=0,
)
try:
    router.completion(model="primary", messages=[{"role": "user", "content": "hi"}], max_tokens=300)
except litellm.NotFoundError as error:
    print(type(error).__name__)
    print(error)

Groq answered with a 404 that says the model does not exist, and LiteLLM raised NotFoundError. The caller has no answer.

Adding a backup group

The Router below has the same broken primary and a working backup. The call asks for primary only. The reply carries headers that say which group answered and how many fallbacks it took.

ExampleAPI keyThe call asks for primary and the backup answers
import litellm
from litellm import Router

router = Router(
    model_list=[
        {"model_name": "primary", "litellm_params": {"model": "groq/this-model-does-not-exist"}},
        {"model_name": "backup", "litellm_params": {"model": "groq/openai/gpt-oss-20b"}},
    ],
    fallbacks=[{"primary": ["backup"]}],
    num_retries=0,
)
response = router.completion(model="primary", messages=[{"role": "user", "content": "Say hi in three words."}],
                             max_tokens=1000)
headers = response._hidden_params["additional_headers"]
print(response.model)
print(response.choices[0].message.content)
print(headers["x-litellm-model-group"], headers["x-litellm-attempted-fallbacks"])
  • response.model is openai/gpt-oss-20b. The model that answered is the deployment behind backup, which proves the fallback fired.
  • x-litellm-model-group is backup. The header names the group that produced the reply, not the group the caller asked for.
  • x-litellm-attempted-fallbacks is 1. The Router moved on once.
  • The caller got an answer. The 404 from the first call never reached the program.

Chaining several fallbacks

A group can list more than one fallback. The Router tries them in order and stops at the first that answers. The two classes below stand in for providers: Down always fails and Up always answers. Each records the deployment it was called as.

python
import litellm
from litellm import CustomLLM, Router
from litellm.llms.custom_llm import CustomLLMError   # what a custom provider raises to fail
python
tried = []                                    # which deployments were called, in order

class Down(CustomLLM):
    def completion(self, model, messages, model_response, **kwargs):
        tried.append(model)
        raise CustomLLMError(status_code=500, message="server down")

class Up(CustomLLM):
    def completion(self, model, messages, model_response, **kwargs):
        tried.append(model)
        model_response.choices[0].message.content = "answered by " + model
        model_response.model = model
        return model_response

Registering them gives the prefixes down/ and up/ a meaning.

python
litellm.custom_provider_map = [
    {"provider": "down", "custom_handler": Down()},
    {"provider": "up", "custom_handler": Up()},
]

The model list gives each its own group. The loop runs the chain twice, with no retries and with one retry, to show how retries and fallbacks combine.

ExampleA chain of two failing groups and one that answers
for retries in [0, 1]:
    tried.clear()
    router = Router(
        model_list=[
            {"model_name": "primary", "litellm_params": {"model": "down/a"}},
            {"model_name": "second", "litellm_params": {"model": "down/b"}},
            {"model_name": "third", "litellm_params": {"model": "up/c"}},
        ],
        fallbacks=[{"primary": ["second", "third"]}],
        num_retries=retries,
    )
    response = router.completion(model="primary", messages=[{"role": "user", "content": "hi"}])
    print("num_retries", retries, tried, response.choices[0].message.content)
  • With no retries the order is a, b, c. Each group was called once, and third answered.
  • With one retry the order is a, a, b, b, c. Each failing group got its retry before the Router moved on, and the answering group was called once.

Other kinds of fallback

  • default_fallbacks is a list of groups tried when any group fails and has no entry of its own in fallbacks.
  • context_window_fallbacks applies when a prompt is too long for the group's model, so the request moves to a group with a larger window.
  • content_policy_fallbacks applies when a provider refuses a request on content grounds.

These are described in the reliability docs and are not run here. Fallbacks and caching in the gateway config writes the same settings in a config file.

Retries vs fallbacks

RetryFallback
What it repeatsThe same groupA different group
Helps withShort failures such as a rate limitA failure that lasts, such as a retired model or an outage
Set withnum_retries or retry_policyfallbacks
OrderFirst, inside the groupAfter the group has used its retries

Where you use fallbacks

  • Provider outages. A Groq group falls back to a Gemini group when Groq is down.
  • Retired models. A deprecated model name keeps answering while you update the list.
  • Long prompts. A context-window fallback moves a long request to a model that can hold it.
Watch out. A fallback answers with a different model, so the reply may differ in style, length and cost from the primary's. Name the group that answered in your logs. The x-litellm-model-group header and response.model tell you, and a quiet fallback that is never noticed can run for weeks.
Try it yourself
  • In the chain example, change up/c to down/c and read the error the Router raises when every group fails.
  • Change the primary in the backup example to a real model, groq/openai/gpt-oss-120b, and print the headers again. No fallback is attempted.
  • Add a second entry to the chain so that second has its own fallback, {"second": ["third"]}, and remove third from the primary list.

Every expert started right here.