Project: running the desk on a morning of tickets
A morning of tickets is the captured run of the desk project: app.py sorts five tickets through the gateway while it falls back, caches and refuses.
Last updated: 12 Oct, 2026 · LiteLLM 1.105
Project: the desk that calls it saved app.py and Project: the gateway a team runs saved config.yaml and desk_guardrail.py. This lesson runs them together for real, nothing changed, and then breaks the gateway twice to see what the app notices. The outputs are what the run printed. Everything here uses the Groq models and the course key, and the replies are live, so your run can differ in the labels and the cost.
Running the app with no gateway
The first mistake is the commonest: the app is started and the gateway is not. The example runs app.py as a child process and prints its exit code and the last line of its error.
import subprocess
import sys
result = subprocess.run([sys.executable, "app.py"], capture_output=True, text=True)
print("exit code:", result.returncode)
print(result.stderr.strip().splitlines()[-1])exit code: 1 openai.APIConnectionError: Connection error.
- The exit code is 1. The program stopped on its first ticket and wrote no ticket lines.
- The last line names the failure.
APIConnectionErroris the OpenAI client's error when no server answers at the address. The app handlesBadRequestErrorfrom a refusal and has no handler for this one, so the error ends the run. A gateway that is down is an outage and not a ticket problem, so stopping is the right behaviour for this app. - The fix is to start the gateway first. The next example does that.
Running the desk against the gateway
Now the gateway is started from the files saved earlier, with QUICK_KEY set to the Groq key, and app.py runs against it. The gateway is stopped in a finally block when the run ends.
import os
import subprocess
import sys
import time
import urllib.request
def start(port=4400, env=None):
proc = subprocess.Popen(
["litellm", "--config", "config.yaml", "--port", str(port)],
stdout=open("gateway.log", "w"), stderr=subprocess.STDOUT, env=env,
)
while True:
try:
urllib.request.urlopen(f"http://127.0.0.1:{port}/health/liveliness")
return proc
except OSError:
time.sleep(1)
proc = start(env={**os.environ, "QUICK_KEY": os.environ["GROQ_API_KEY"]})
try:
result = subprocess.run([sys.executable, "app.py"], capture_output=True, text=True)
print(result.stdout.strip())
if result.returncode:
print(result.stderr.strip().splitlines()[-1])
finally:
proc.terminate()
proc.wait()I was charged twice for one order billing quick 0.000024 My parcel has not arrived shipping quick 0.000019 How do I change my password? refused: The desk never handles passwords. Use the reset link. Do you sell gift cards? other careful 0.000092 I was charged twice for one order billing cache 0.000000 spent 0.000135
- Tickets one and two were labelled by
quick. The third column names the group the gateway says answered, and the cost is the call's cost from the response header. - The password ticket was refused. The line shows the guardrail's own message, which came back as a 400. The app printed it and went on to the next ticket. No model was called for it.
- The gift card ticket was escalated.
quickansweredother, which is not one of the three labels, so the app askedcareful. The group shown iscarefuland the cost is both calls added together, which is why it is larger than the others. - The repeated ticket came from the cache. The group column says
cacheand the app counts it as zero. No model was called for it. - The last line is the total. It is the sum of the costs the headers reported. Labels and costs are live and can differ on your run.
Running it again with a bad key for quick
The second drill spoils the key that only the quick group reads. The app, the config and the tickets are the same. A new gateway starts, so its cache is empty.
import os
import subprocess
import sys
import time
import urllib.request
def start(port=4400, env=None):
proc = subprocess.Popen(
["litellm", "--config", "config.yaml", "--port", str(port)],
stdout=open("gateway.log", "w"), stderr=subprocess.STDOUT, env=env,
)
while True:
try:
urllib.request.urlopen(f"http://127.0.0.1:{port}/health/liveliness")
return proc
except OSError:
time.sleep(1)
proc = start(env={**os.environ, "QUICK_KEY": "not-a-real-key"})
try:
result = subprocess.run([sys.executable, "app.py"], capture_output=True, text=True)
print(result.stdout.strip())
if result.returncode:
print(result.stderr.strip().splitlines()[-1])
finally:
proc.terminate()
proc.wait()I was charged twice for one order billing careful 0.000044 My parcel has not arrived shipping careful 0.000043 How do I change my password? refused: The desk never handles passwords. Use the reset link. Do you sell gift cards? other cache 0.000044 I was charged twice for one order billing cache 0.000000 spent 0.000131
- The app printed no error. Nothing in its output says
quickfailed. The gateway rejected the key at the provider and used the fallback tocareful, and the app only reads the answer. - The group column is the only sign. Tickets one and two now say
careful, and each cost more than the same ticket in the first run. A team watching these headers sees the outage at once. A team that does not sees it as a larger bill. - The guardrail still refused the password ticket. It runs before any model, so a broken model key does not affect it.
- The gift card ticket cost one call and was found in the cache the second time. The first request for
quickwas answered bycarefulthrough the fallback and saidother. The app then askedcareful, and that answer came from the cache becausecarefulhad answered the same prompt a moment before. The cost shown is the first call's. - The repeat was cached again. The new gateway started with an empty cache, and the repeated ticket was found in it after the first ticket was answered.
Normal run vs spoiled key
| Ticket | Normal run: served by and cost | Spoiled key: served by and cost |
|---|---|---|
| I was charged twice for one order | quick, 0.000024 | careful, 0.000044 |
| My parcel has not arrived | quick, 0.000019 | careful, 0.000043 |
| How do I change my password? | refused by the guardrail | refused by the guardrail |
| Do you sell gift cards? | careful after quick said other, 0.000092 | cache, 0.000044 |
| I was charged twice for one order (repeat) | cache, 0 | cache, 0 |
What this course left out
The LiteLLM documentation is larger than one course. These areas are named in it and are not taught here. Some need a database or a service that is not run in this course, and the rest are further reading.
| Area in the docs | Why it is not taught here |
|---|---|
| Teams, organisations and end users | Needs the database that virtual keys need |
| Enforced budgets, per-model budgets | Needs the database; the budgets page documents them |
| Single sign-on and JWT for the dashboard | Needs the dashboard and an identity provider |
| Vendor guardrails and Presidio | Need accounts, services or packages not set up here |
| Redis and other cache types | Need a server this course does not run |
| Pass-through endpoints, batches, files and fine-tuning | Further reading in the proxy section of the docs |
| Prompt management | Further reading in the proxy section of the docs |
| Running at scale on Kubernetes | Needs a cluster |
| Provider pages | One page per provider in the docs, listing the arguments that provider takes |
Where a run like this fits
- A rehearsal before a deployment. Break the key for one group and read what the app reports.
- A check after a config change. The same five tickets give a before and an after.
- An example for a teammate. The output shows the gateway's behaviour without opening the dashboard.
Related
- Previous: Project: the desk that calls it
- See also: Hybrid SDK, Router and gateway for when to use the SDK, the Router or the gateway
- Reference: Reliability in the LiteLLM docs
- Run the first example after starting the gateway by hand with
litellm --config config.yaml --port 4400, and stop it with Ctrl+C when done. - Add a ticket that mentions a password in other words, and see whether the guardrail catches it.
- Start the gateway with the real
QUICK_KEYand a badGROQ_API_KEY, run the desk, and read the error.carefulhas no fallback entry, and the gateway's message says so.
Every expert started right here.