Fallbacks and caching in the gateway config
A gateway config is the config.yaml where fallbacks, retries and caching are declared once, so that every application calling the gateway gets them without changing its code.
Last updated: 12 Oct, 2026 · LiteLLM 1.105
Fallbacks: another model group when one fails, Retries: trying again when a provider refuses and Caching responses each set these up in Python, in one program. MCP tools through the gateway ended the tour of gateway features. This lesson puts fallbacks and caching in the file the gateway reads, and runs them against a real gateway: a first model group that cannot be reached, a fallback to Groq, and an identical second request answered from the cache.
Syntax: fallbacks go under router_settings as a list of dictionaries, each mapping a model group to the groups to try next. The reliability page shows a fallback as a mapping from a model group to a list of groups, and says fallbacks are tried in order. It puts fallbacks under router_settings in its quick start and under litellm_settings in its advanced examples, and does not say that one is required. This course uses router_settings. Caching goes under litellm_settings, as the caching page shows.
router_settings:
fallbacks: [{"quick": ["careful"]}] # if quick fails, try careful
litellm_settings:
cache: true
cache_params:
type: local # an in-memory cacheDeclaring a model group that fails
A fallback needs something to fall back from. The group quick points at http://127.0.0.1:9, an address nothing listens on, so every call to it fails with a connection error. careful is the real Groq model. In a deployment quick would be a real model that sometimes fails, and the config would read the same.
model_list:
- model_name: quick
litellm_params:
model: openai/stub
api_base: http://127.0.0.1:9 # nothing listens here
api_key: none
- model_name: careful
litellm_params:
model: groq/openai/gpt-oss-120b
api_key: os.environ/GROQ_API_KEYAsking for the failing group twice
The example writes the whole config, starts a gateway, and sends the same request for quick twice with the OpenAI client. For each answer it prints the group that served it, the number of fallbacks tried and whether a cache key header is present.
import subprocess
import time
import urllib.request
from pathlib import Path
from openai import OpenAI
def start(port=4400, env=None):
proc = subprocess.Popen(
["litellm", "--config", "config.yaml", "--port", str(port)],
stdout=open("gateway.log", "w"), stderr=subprocess.STDOUT, env=env,
)
while True:
try:
urllib.request.urlopen(f"http://127.0.0.1:{port}/health/liveliness")
return proc
except OSError:
time.sleep(1)
Path("config.yaml").write_text("""model_list:
- model_name: quick
litellm_params:
model: openai/stub
api_base: http://127.0.0.1:9
api_key: none
- model_name: careful
litellm_params:
model: groq/openai/gpt-oss-120b
api_key: os.environ/GROQ_API_KEY
litellm_settings:
cache: true
cache_params:
type: local
router_settings:
fallbacks: [{"quick": ["careful"]}]
general_settings:
master_key: sk-gateway-1234
""")
proc = start()
try:
client = OpenAI(base_url="http://127.0.0.1:4400", api_key="sk-gateway-1234")
for attempt in (1, 2):
raw = client.chat.completions.with_raw_response.create(
model="quick", max_tokens=200,
messages=[{"role": "user", "content": "Reply with one word: ok"}],
)
h = raw.headers
print(attempt, raw.parse().choices[0].message.content)
print(" served by:", h["x-litellm-model-group"], "| fallbacks:", h["x-litellm-attempted-fallbacks"],
"| from cache:", "x-litellm-cache-key" in h)
finally:
proc.terminate()
proc.wait()1 ok served by: careful | fallbacks: 1 | from cache: False 2 ok served by: careful | fallbacks: 1 | from cache: True
- The first answer came from
careful. The application asked forquick, the gateway failed to reach it and triedcareful, so the served group iscarefuland one fallback was attempted. - The second answer came from the cache. The same request had been answered, so the gateway returned the stored answer and added an
x-litellm-cache-keyheader. The caching page says a cached response carries that header. - The text is a live model reply. It can differ on your run, and the second line repeats the first because it was stored.
- The application did nothing. It sent two ordinary requests for a model name, with no retry code and no cache code.
What the config does not show
The cache here is type: local, which holds answers in the gateway process's memory, so it is empty after a restart and is not shared between several gateway containers. The caching page lists other types in its cache_params.type table, including redis for Redis and similar services. A Redis cache is not run here, because it needs a Redis server. Retries are a different setting. The reliability page puts num_retries under litellm_settings in its example, and says that a call which still fails after its retries moves to the fallback. The project in this part sets num_retries there, and not under router_settings.
Python settings vs config.yaml
| In your Python code | In config.yaml | |
|---|---|---|
| Fallbacks | Router(fallbacks=[...]) in each program | router_settings.fallbacks once |
| Caching | litellm.cache in each program | litellm_settings.cache once |
| Who gets it | The program that sets it | Every application calling the gateway |
| Changing it | Edit and redeploy each program | Edit the file and restart the gateway |
Where gateway config fits
- A provider outage. The fallback sends traffic to another group while the first recovers, and applications notice only through the headers.
- Repeated questions. A support desk sees the same ticket again and again, and the cache answers the repeats without a model call.
- One change for many programs. Swapping the fallback target is a one-line edit to the file.
x-litellm-attempted-fallbacks, the first sign of an outage is the provider's bill for the more expensive group. Log the header, as the observability lesson showed.Related
- Previous: MCP tools through the gateway
- Next: Project: the gateway a team runs
- Reference: Reliability and Caching in the LiteLLM docs
- Change the message in the second request by one word and check whether the cache key header appears.
- Remove the
fallbacksline, restart the gateway and send the request again. Read the error. - Point
quickatgroq/openai/gpt-oss-20bwith your Groq key and check which group serves the first request.
You understood something today that you didn't yesterday.