LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

Project: the gateway a team runs

The desk gateway is the server in this project: one config.yaml that names two model groups, a retry, a fallback between them, a cache, a guardrail and a master key.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

Fallbacks and caching in the gateway config put fallbacks and caching in a config file. This project adds the guardrail from Guardrails on every request and builds a small support desk around the result. The desk sorts customer tickets into billing, shipping and account questions. A cheap model, quick, tries each ticket first, and a larger one, careful, takes over when quick fails or cannot decide. The project runs on a gateway with no database. Virtual keys, budgets and the admin dashboard need Postgres, which this course does not run, so the desk app uses the master key. The project says where each of those would go and does not pretend to use them.

The shape of the desk project. The desk app, app.py, sends each ticket to a gateway started from config.yaml. Inside the gateway the request meets a no-passwords guardrail, which refuses password tickets with an error returned to the app, then a cache, then the quick model group backed by gpt-oss-20b. A dashed arrow shows a fallback from quick to the careful model group backed by gpt-oss-120b when quick fails.

Naming the two model groups

A model group is a name an application asks for. Each entry in model_list gives the group a model_name and the provider model behind it. quick is the small Groq model, gpt-oss-20b, and careful is the larger gpt-oss-120b. The two read different environment variables for their key, QUICK_KEY and GROQ_API_KEY. In normal use both hold the same Groq key. They are separate so that a later lesson can spoil the key for quick alone.

yaml
model_list:
  - model_name: quick
    litellm_params:
      model: groq/openai/gpt-oss-20b
      api_key: os.environ/QUICK_KEY
  - model_name: careful
    litellm_params:
      model: groq/openai/gpt-oss-120b
      api_key: os.environ/GROQ_API_KEY

Declaring retries, the fallback and the cache

num_retries: 1 under litellm_settings asks for one more try on a failing call, the placement the reliability page shows. cache: true with cache_params.type: local keeps answers in the gateway's memory. fallbacks under router_settings sends a failing quick request to careful.

yaml
litellm_settings:
  num_retries: 1
  cache: true
  cache_params:
    type: local

router_settings:
  fallbacks: [{"quick": ["careful"]}]

Adding the guardrail and the master key

The guardrail is the NoPasswords class from Guardrails on every request, saved as desk_guardrail.py next to the config. It refuses any ticket that mentions a password before a model is called. master_key is the one key an application sends, and default_on: true applies the guardrail to every request.

yaml
guardrails:
  - guardrail_name: no-passwords
    litellm_params:
      guardrail: desk_guardrail.NoPasswords
      mode: pre_call
      default_on: true

general_settings:
  master_key: sk-gateway-1234

Starting the desk gateway

The example saves desk_guardrail.py and the complete config.yaml, which are the project's files. It starts the gateway with QUICK_KEY set to the Groq key and asks which models it serves. These two files do not change again in the project.

ExampleThe desk gateway, started from its config
import os
import subprocess
import time
import urllib.request
from pathlib import Path

import httpx


def start(port=4400, env=None):
    proc = subprocess.Popen(
        ["litellm", "--config", "config.yaml", "--port", str(port)],
        stdout=open("gateway.log", "w"), stderr=subprocess.STDOUT, env=env,
    )
    while True:
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{port}/health/liveliness")
            return proc
        except OSError:
            time.sleep(1)


Path("desk_guardrail.py").write_text(r'''from fastapi import HTTPException
from litellm.integrations.custom_guardrail import CustomGuardrail


class NoPasswords(CustomGuardrail):
    """Refuses any request that mentions a password."""

    async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type):
        for message in data.get("messages", []):
            if "password" in str(message.get("content", "")).lower():
                raise HTTPException(
                    status_code=400,
                    detail={"error": "The desk never handles passwords. Use the reset link."},
                )
        return data
''')
Path("config.yaml").write_text("""model_list:
  - model_name: quick
    litellm_params:
      model: groq/openai/gpt-oss-20b
      api_key: os.environ/QUICK_KEY
  - model_name: careful
    litellm_params:
      model: groq/openai/gpt-oss-120b
      api_key: os.environ/GROQ_API_KEY

litellm_settings:
  num_retries: 1
  cache: true
  cache_params:
    type: local

router_settings:
  fallbacks: [{"quick": ["careful"]}]

guardrails:
  - guardrail_name: no-passwords
    litellm_params:
      guardrail: desk_guardrail.NoPasswords
      mode: pre_call
      default_on: true

general_settings:
  master_key: sk-gateway-1234
""")

proc = start(env={**os.environ, "QUICK_KEY": os.environ["GROQ_API_KEY"]})
try:
    r = httpx.get("http://127.0.0.1:4400/v1/models", headers={"Authorization": "Bearer sk-gateway-1234"})
    print(r.status_code, [m["id"] for m in r.json()["data"]])
finally:
    proc.terminate()
    proc.wait()
  • Both groups are served. The gateway lists quick and careful, the two names an application will ask for.
  • The guardrail loaded. A gateway that cannot import desk_guardrail.NoPasswords does not come up, and this one did, so the file and the class path agree.
  • The cache, the retry and the fallback are in force. They show their effect on requests, which the last lesson of the project runs.

What each setting came from

SettingWhat it does hereWhere it was taught
model_listTwo named groups on GroqRouter: model groups and deployments
fallbacksquick fails, careful answersFallbacks: another model group when one fails
num_retriesOne more try on a failing callRetries: trying again when a provider refuses
cacheRepeated tickets are not paid for twiceCaching responses
guardrailsPassword tickets never reach a modelGuardrails on every request
master_keyThe one key the desk app sendsGateway keys

What would be added with a database

  • A virtual key for the desk app. The app would hold its own key, with a budget, instead of the master key, as the virtual keys lesson showed.
  • The dashboard. Logs and Usage would show each ticket's cost against the app's key.
  • A callback or Langfuse. Every call could be forwarded to a tracing tool, as the observability lesson showed.
Watch out. master_key: sk-gateway-1234 is a teaching value and it sits in the file. In a deployment, give the gateway the key through LITELLM_MASTER_KEY and keep it out of version control, as the container lesson showed.
Try it yourself
  • Add a third group named cheap with the same model as quick and list the models again.
  • Change the guardrail's refusal text in desk_guardrail.py and start the gateway again.
  • Change num_retries to 3, run the example again and check that the same two models are listed.

Slow is fine. Stopping is the only problem.