LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

Project: running the desk on a morning of tickets

A morning of tickets is the captured run of the desk project: app.py sorts five tickets through the gateway while it falls back, caches and refuses.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

Project: the desk that calls it saved app.py and Project: the gateway a team runs saved config.yaml and desk_guardrail.py. This lesson runs them together for real, nothing changed, and then breaks the gateway twice to see what the app notices. The outputs are what the run printed. Everything here uses the Groq models and the course key, and the replies are live, so your run can differ in the labels and the cost.

Running the app with no gateway

The first mistake is the commonest: the app is started and the gateway is not. The example runs app.py as a child process and prints its exit code and the last line of its error.

ExampleThe app run before the gateway is started
import subprocess
import sys

result = subprocess.run([sys.executable, "app.py"], capture_output=True, text=True)
print("exit code:", result.returncode)
print(result.stderr.strip().splitlines()[-1])
  • The exit code is 1. The program stopped on its first ticket and wrote no ticket lines.
  • The last line names the failure. APIConnectionError is the OpenAI client's error when no server answers at the address. The app handles BadRequestError from a refusal and has no handler for this one, so the error ends the run. A gateway that is down is an outage and not a ticket problem, so stopping is the right behaviour for this app.
  • The fix is to start the gateway first. The next example does that.

Running the desk against the gateway

Now the gateway is started from the files saved earlier, with QUICK_KEY set to the Groq key, and app.py runs against it. The gateway is stopped in a finally block when the run ends.

ExampleAPI keyThe desk, run for real
import os
import subprocess
import sys
import time
import urllib.request


def start(port=4400, env=None):
    proc = subprocess.Popen(
        ["litellm", "--config", "config.yaml", "--port", str(port)],
        stdout=open("gateway.log", "w"), stderr=subprocess.STDOUT, env=env,
    )
    while True:
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{port}/health/liveliness")
            return proc
        except OSError:
            time.sleep(1)



proc = start(env={**os.environ, "QUICK_KEY": os.environ["GROQ_API_KEY"]})
try:
    result = subprocess.run([sys.executable, "app.py"], capture_output=True, text=True)
    print(result.stdout.strip())
    if result.returncode:
        print(result.stderr.strip().splitlines()[-1])
finally:
    proc.terminate()
    proc.wait()
  • Tickets one and two were labelled by quick. The third column names the group the gateway says answered, and the cost is the call's cost from the response header.
  • The password ticket was refused. The line shows the guardrail's own message, which came back as a 400. The app printed it and went on to the next ticket. No model was called for it.
  • The gift card ticket was escalated. quick answered other, which is not one of the three labels, so the app asked careful. The group shown is careful and the cost is both calls added together, which is why it is larger than the others.
  • The repeated ticket came from the cache. The group column says cache and the app counts it as zero. No model was called for it.
  • The last line is the total. It is the sum of the costs the headers reported. Labels and costs are live and can differ on your run.

Running it again with a bad key for quick

The second drill spoils the key that only the quick group reads. The app, the config and the tickets are the same. A new gateway starts, so its cache is empty.

ExampleAPI keyThe same desk with a rejected key for quick
import os
import subprocess
import sys
import time
import urllib.request


def start(port=4400, env=None):
    proc = subprocess.Popen(
        ["litellm", "--config", "config.yaml", "--port", str(port)],
        stdout=open("gateway.log", "w"), stderr=subprocess.STDOUT, env=env,
    )
    while True:
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{port}/health/liveliness")
            return proc
        except OSError:
            time.sleep(1)



proc = start(env={**os.environ, "QUICK_KEY": "not-a-real-key"})
try:
    result = subprocess.run([sys.executable, "app.py"], capture_output=True, text=True)
    print(result.stdout.strip())
    if result.returncode:
        print(result.stderr.strip().splitlines()[-1])
finally:
    proc.terminate()
    proc.wait()
  • The app printed no error. Nothing in its output says quick failed. The gateway rejected the key at the provider and used the fallback to careful, and the app only reads the answer.
  • The group column is the only sign. Tickets one and two now say careful, and each cost more than the same ticket in the first run. A team watching these headers sees the outage at once. A team that does not sees it as a larger bill.
  • The guardrail still refused the password ticket. It runs before any model, so a broken model key does not affect it.
  • The gift card ticket cost one call and was found in the cache the second time. The first request for quick was answered by careful through the fallback and said other. The app then asked careful, and that answer came from the cache because careful had answered the same prompt a moment before. The cost shown is the first call's.
  • The repeat was cached again. The new gateway started with an empty cache, and the repeated ticket was found in it after the first ticket was answered.

Normal run vs spoiled key

TicketNormal run: served by and costSpoiled key: served by and cost
I was charged twice for one orderquick, 0.000024careful, 0.000044
My parcel has not arrivedquick, 0.000019careful, 0.000043
How do I change my password?refused by the guardrailrefused by the guardrail
Do you sell gift cards?careful after quick said other, 0.000092cache, 0.000044
I was charged twice for one order (repeat)cache, 0cache, 0

What this course left out

The LiteLLM documentation is larger than one course. These areas are named in it and are not taught here. Some need a database or a service that is not run in this course, and the rest are further reading.

Area in the docsWhy it is not taught here
Teams, organisations and end usersNeeds the database that virtual keys need
Enforced budgets, per-model budgetsNeeds the database; the budgets page documents them
Single sign-on and JWT for the dashboardNeeds the dashboard and an identity provider
Vendor guardrails and PresidioNeed accounts, services or packages not set up here
Redis and other cache typesNeed a server this course does not run
Pass-through endpoints, batches, files and fine-tuningFurther reading in the proxy section of the docs
Prompt managementFurther reading in the proxy section of the docs
Running at scale on KubernetesNeeds a cluster
Provider pagesOne page per provider in the docs, listing the arguments that provider takes

Where a run like this fits

  • A rehearsal before a deployment. Break the key for one group and read what the app reports.
  • A check after a config change. The same five tickets give a before and an after.
  • An example for a teammate. The output shows the gateway's behaviour without opening the dashboard.
Watch out. The run proves the fallback path on one failure type, a rejected key, and the cache on one repeated ticket. It does not prove a budget, a virtual key or the dashboard, which need a database that is not run here.
Try it yourself
  • Run the first example after starting the gateway by hand with litellm --config config.yaml --port 4400, and stop it with Ctrl+C when done.
  • Add a ticket that mentions a password in other words, and see whether the guardrail catches it.
  • Start the gateway with the real QUICK_KEY and a bad GROQ_API_KEY, run the desk, and read the error. careful has no fallback entry, and the gateway's message says so.

Every expert started right here.