Project: the gateway a team runs
The desk gateway is the server in this project: one config.yaml that names two model groups, a retry, a fallback between them, a cache, a guardrail and a master key.
Last updated: 12 Oct, 2026 · LiteLLM 1.105
Fallbacks and caching in the gateway config put fallbacks and caching in a config file. This project adds the guardrail from Guardrails on every request and builds a small support desk around the result. The desk sorts customer tickets into billing, shipping and account questions. A cheap model, quick, tries each ticket first, and a larger one, careful, takes over when quick fails or cannot decide. The project runs on a gateway with no database. Virtual keys, budgets and the admin dashboard need Postgres, which this course does not run, so the desk app uses the master key. The project says where each of those would go and does not pretend to use them.
Naming the two model groups
A model group is a name an application asks for. Each entry in model_list gives the group a model_name and the provider model behind it. quick is the small Groq model, gpt-oss-20b, and careful is the larger gpt-oss-120b. The two read different environment variables for their key, QUICK_KEY and GROQ_API_KEY. In normal use both hold the same Groq key. They are separate so that a later lesson can spoil the key for quick alone.
model_list:
- model_name: quick
litellm_params:
model: groq/openai/gpt-oss-20b
api_key: os.environ/QUICK_KEY
- model_name: careful
litellm_params:
model: groq/openai/gpt-oss-120b
api_key: os.environ/GROQ_API_KEYDeclaring retries, the fallback and the cache
num_retries: 1 under litellm_settings asks for one more try on a failing call, the placement the reliability page shows. cache: true with cache_params.type: local keeps answers in the gateway's memory. fallbacks under router_settings sends a failing quick request to careful.
litellm_settings:
num_retries: 1
cache: true
cache_params:
type: local
router_settings:
fallbacks: [{"quick": ["careful"]}]Adding the guardrail and the master key
The guardrail is the NoPasswords class from Guardrails on every request, saved as desk_guardrail.py next to the config. It refuses any ticket that mentions a password before a model is called. master_key is the one key an application sends, and default_on: true applies the guardrail to every request.
guardrails:
- guardrail_name: no-passwords
litellm_params:
guardrail: desk_guardrail.NoPasswords
mode: pre_call
default_on: true
general_settings:
master_key: sk-gateway-1234Starting the desk gateway
The example saves desk_guardrail.py and the complete config.yaml, which are the project's files. It starts the gateway with QUICK_KEY set to the Groq key and asks which models it serves. These two files do not change again in the project.
import os
import subprocess
import time
import urllib.request
from pathlib import Path
import httpx
def start(port=4400, env=None):
proc = subprocess.Popen(
["litellm", "--config", "config.yaml", "--port", str(port)],
stdout=open("gateway.log", "w"), stderr=subprocess.STDOUT, env=env,
)
while True:
try:
urllib.request.urlopen(f"http://127.0.0.1:{port}/health/liveliness")
return proc
except OSError:
time.sleep(1)
Path("desk_guardrail.py").write_text(r'''from fastapi import HTTPException
from litellm.integrations.custom_guardrail import CustomGuardrail
class NoPasswords(CustomGuardrail):
"""Refuses any request that mentions a password."""
async def async_pre_call_hook(self, user_api_key_dict, cache, data, call_type):
for message in data.get("messages", []):
if "password" in str(message.get("content", "")).lower():
raise HTTPException(
status_code=400,
detail={"error": "The desk never handles passwords. Use the reset link."},
)
return data
''')
Path("config.yaml").write_text("""model_list:
- model_name: quick
litellm_params:
model: groq/openai/gpt-oss-20b
api_key: os.environ/QUICK_KEY
- model_name: careful
litellm_params:
model: groq/openai/gpt-oss-120b
api_key: os.environ/GROQ_API_KEY
litellm_settings:
num_retries: 1
cache: true
cache_params:
type: local
router_settings:
fallbacks: [{"quick": ["careful"]}]
guardrails:
- guardrail_name: no-passwords
litellm_params:
guardrail: desk_guardrail.NoPasswords
mode: pre_call
default_on: true
general_settings:
master_key: sk-gateway-1234
""")
proc = start(env={**os.environ, "QUICK_KEY": os.environ["GROQ_API_KEY"]})
try:
r = httpx.get("http://127.0.0.1:4400/v1/models", headers={"Authorization": "Bearer sk-gateway-1234"})
print(r.status_code, [m["id"] for m in r.json()["data"]])
finally:
proc.terminate()
proc.wait()200 ['quick', 'careful']
- Both groups are served. The gateway lists
quickandcareful, the two names an application will ask for. - The guardrail loaded. A gateway that cannot import
desk_guardrail.NoPasswordsdoes not come up, and this one did, so the file and the class path agree. - The cache, the retry and the fallback are in force. They show their effect on requests, which the last lesson of the project runs.
What each setting came from
| Setting | What it does here | Where it was taught |
|---|---|---|
model_list | Two named groups on Groq | Router: model groups and deployments |
fallbacks | quick fails, careful answers | Fallbacks: another model group when one fails |
num_retries | One more try on a failing call | Retries: trying again when a provider refuses |
cache | Repeated tickets are not paid for twice | Caching responses |
guardrails | Password tickets never reach a model | Guardrails on every request |
master_key | The one key the desk app sends | Gateway keys |
What would be added with a database
- A virtual key for the desk app. The app would hold its own key, with a budget, instead of the master key, as the virtual keys lesson showed.
- The dashboard. Logs and Usage would show each ticket's cost against the app's key.
- A callback or Langfuse. Every call could be forwarded to a tracing tool, as the observability lesson showed.
master_key: sk-gateway-1234 is a teaching value and it sits in the file. In a deployment, give the gateway the key through LITELLM_MASTER_KEY and keep it out of version control, as the container lesson showed.Related
- Previous: Fallbacks and caching in the gateway config
- Next: Project: the desk that calls it
- Reference: Proxy config.yaml in the LiteLLM docs
- Add a third group named
cheapwith the same model asquickand list the models again. - Change the guardrail's refusal text in
desk_guardrail.pyand start the gateway again. - Change
num_retriesto 3, run the example again and check that the same two models are listed.
Slow is fine. Stopping is the only problem.