LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

Fallbacks and caching in the gateway config

A gateway config is the config.yaml where fallbacks, retries and caching are declared once, so that every application calling the gateway gets them without changing its code.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

Fallbacks: another model group when one fails, Retries: trying again when a provider refuses and Caching responses each set these up in Python, in one program. MCP tools through the gateway ended the tour of gateway features. This lesson puts fallbacks and caching in the file the gateway reads, and runs them against a real gateway: a first model group that cannot be reached, a fallback to Groq, and an identical second request answered from the cache.

The path of one request through a gateway configured with config.yaml. The app's request first passes the master key check, then a cache lookup. On a cache hit a dashed arrow returns the stored answer to the app without calling a provider. On a miss the request goes to the quick model group, and a dashed arrow shows a fallback to the careful model group when quick fails. Both groups call the model providers.

Syntax: fallbacks go under router_settings as a list of dictionaries, each mapping a model group to the groups to try next. The reliability page shows a fallback as a mapping from a model group to a list of groups, and says fallbacks are tried in order. It puts fallbacks under router_settings in its quick start and under litellm_settings in its advanced examples, and does not say that one is required. This course uses router_settings. Caching goes under litellm_settings, as the caching page shows.

yaml
router_settings:
  fallbacks: [{"quick": ["careful"]}]   # if quick fails, try careful

litellm_settings:
  cache: true
  cache_params:
    type: local                         # an in-memory cache

Declaring a model group that fails

A fallback needs something to fall back from. The group quick points at http://127.0.0.1:9, an address nothing listens on, so every call to it fails with a connection error. careful is the real Groq model. In a deployment quick would be a real model that sometimes fails, and the config would read the same.

yaml
model_list:
  - model_name: quick
    litellm_params:
      model: openai/stub
      api_base: http://127.0.0.1:9    # nothing listens here
      api_key: none
  - model_name: careful
    litellm_params:
      model: groq/openai/gpt-oss-120b
      api_key: os.environ/GROQ_API_KEY

Asking for the failing group twice

The example writes the whole config, starts a gateway, and sends the same request for quick twice with the OpenAI client. For each answer it prints the group that served it, the number of fallbacks tried and whether a cache key header is present.

ExampleAPI keyA fallback and a cache hit in a real gateway
import subprocess
import time
import urllib.request
from pathlib import Path

from openai import OpenAI


def start(port=4400, env=None):
    proc = subprocess.Popen(
        ["litellm", "--config", "config.yaml", "--port", str(port)],
        stdout=open("gateway.log", "w"), stderr=subprocess.STDOUT, env=env,
    )
    while True:
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{port}/health/liveliness")
            return proc
        except OSError:
            time.sleep(1)


Path("config.yaml").write_text("""model_list:
  - model_name: quick
    litellm_params:
      model: openai/stub
      api_base: http://127.0.0.1:9
      api_key: none
  - model_name: careful
    litellm_params:
      model: groq/openai/gpt-oss-120b
      api_key: os.environ/GROQ_API_KEY

litellm_settings:
  cache: true
  cache_params:
    type: local

router_settings:
  fallbacks: [{"quick": ["careful"]}]

general_settings:
  master_key: sk-gateway-1234
""")

proc = start()
try:
    client = OpenAI(base_url="http://127.0.0.1:4400", api_key="sk-gateway-1234")
    for attempt in (1, 2):
        raw = client.chat.completions.with_raw_response.create(
            model="quick", max_tokens=200,
            messages=[{"role": "user", "content": "Reply with one word: ok"}],
        )
        h = raw.headers
        print(attempt, raw.parse().choices[0].message.content)
        print("   served by:", h["x-litellm-model-group"], "| fallbacks:", h["x-litellm-attempted-fallbacks"],
              "| from cache:", "x-litellm-cache-key" in h)
finally:
    proc.terminate()
    proc.wait()
  • The first answer came from careful. The application asked for quick, the gateway failed to reach it and tried careful, so the served group is careful and one fallback was attempted.
  • The second answer came from the cache. The same request had been answered, so the gateway returned the stored answer and added an x-litellm-cache-key header. The caching page says a cached response carries that header.
  • The text is a live model reply. It can differ on your run, and the second line repeats the first because it was stored.
  • The application did nothing. It sent two ordinary requests for a model name, with no retry code and no cache code.

What the config does not show

The cache here is type: local, which holds answers in the gateway process's memory, so it is empty after a restart and is not shared between several gateway containers. The caching page lists other types in its cache_params.type table, including redis for Redis and similar services. A Redis cache is not run here, because it needs a Redis server. Retries are a different setting. The reliability page puts num_retries under litellm_settings in its example, and says that a call which still fails after its retries moves to the fallback. The project in this part sets num_retries there, and not under router_settings.

Python settings vs config.yaml

In your Python codeIn config.yaml
FallbacksRouter(fallbacks=[...]) in each programrouter_settings.fallbacks once
Cachinglitellm.cache in each programlitellm_settings.cache once
Who gets itThe program that sets itEvery application calling the gateway
Changing itEdit and redeploy each programEdit the file and restart the gateway

Where gateway config fits

  • A provider outage. The fallback sends traffic to another group while the first recovers, and applications notice only through the headers.
  • Repeated questions. A support desk sees the same ticket again and again, and the cache answers the repeats without a model call.
  • One change for many programs. Swapping the fallback target is a one-line edit to the file.
Watch out. A fallback hides a failing model. If nobody reads x-litellm-attempted-fallbacks, the first sign of an outage is the provider's bill for the more expensive group. Log the header, as the observability lesson showed.
Try it yourself
  • Change the message in the second request by one word and check whether the cache key header appears.
  • Remove the fallbacks line, restart the gateway and send the request again. Read the error.
  • Point quick at groq/openai/gpt-oss-20b with your Groq key and check which group serves the first request.

You understood something today that you didn't yesterday.