Fallbacks and caching in the gateway config

Everything from parts 3 and 4 moves into config.yaml, where it applies to every app at once, and the gateway reports on each answer what it did.

Exampleconfig.yaml

model_list:
  - model_name: careful
    litellm_params:
      model: down/large
  - model_name: quick
    litellm_params:
      model: shop/small

litellm_settings:
  custom_provider_map:
    - {"provider": "shop", "custom_handler": flaky_llm.shop}
    - {"provider": "down", "custom_handler": flaky_llm.down}
  cache: true
  cache_params:
    type: local

router_settings:
  fallbacks: [{"careful": ["quick"]}]
  num_retries: 0

general_settings:
  master_key: sk-gateway-1234

Two groups: careful is the provider that is down, quick the one that works. router_settings holds the Router's fallbacks and retries from lessons 9 to 11. cache: true with type: local is the in-memory cache from lesson 13; the gateway caching page lists Redis, S3 and others for real deployments. The handlers come from flaky_llm.py.

What the gateway tells you

Exampleapp.py

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:4000", api_key="sk-gateway-1234")

for attempt in (1, 2):
    raw = client.chat.completions.with_raw_response.create(
        model="careful",
        messages=[{"role": "user", "content": "I was charged twice for one order"}],
    )
    reply = raw.parse()
    print(attempt, reply.choices[0].message.content)
    print("   group:", raw.headers["x-litellm-model-group"],
          "| fallbacks tried:", raw.headers["x-litellm-attempted-fallbacks"],
          "| cached:", "x-litellm-cache-key" in raw.headers)

with_raw_response is the openai package's way to see the HTTP response behind a call; parse() gives the normal reply. The gateway adds x-litellm- headers describing how it served the request.

Example

litellm --config config.yaml --port 4000 > gateway.log 2>&1 &
until curl -s localhost:4000/health/liveliness > /dev/null; do sleep 1; done
python app.py
kill %1

The app asked for careful both times. The first answer came from the quick group after one fallback. The second was identical and came from the cache, which is what the cache key header marks. The app's code handled none of this.

Those headers, together with gateway.log, are what to read when someone asks why an answer was worse today or cheaper than expected.

Try it yourself

Remove the fallbacks line, restart, and run app.py.
Print raw.headers["x-litellm-response-cost"] if your version sends it, or list every x-litellm- header.
Change the second message by one word and check whether it is cached.

PreviousKeys: who may call the gateway

Slow is fine. Stopping is the only problem.