Project: the desk that calls it
The desk app is a short Python program that sends support tickets to the gateway with the OpenAI client and sorts each one into billing, shipping or account.
Last updated: 12 Oct, 2026 · LiteLLM 1.105
Project: the gateway a team runs started the server. The app is its client, and it holds none of the gateway's work: no provider key, no retry loop, no cache and no model names beyond the two groups. This lesson builds the file, app.py, in pieces and then saves it. This is the one version of app.py in the project. Project: running the desk on a morning of tickets runs it unchanged, and if a change is ever needed, that change is made to this file and stated.
Connecting to the gateway
The OpenAI client works against the gateway because the gateway speaks the same API. Only base_url and the key change: base_url points at the gateway's address and api_key is the master key from the config.
from openai import BadRequestError, OpenAI
client = OpenAI(base_url="http://127.0.0.1:4400", api_key="sk-gateway-1234")Listing the tickets and the question
Five tickets stand for a morning's queue. One mentions a password, which the guardrail will refuse. One does not fit the three labels. The first ticket comes back at the end, as repeats do on a real desk. The prompt asks for one word.
TICKETS = [
"I was charged twice for one order",
"My parcel has not arrived",
"How do I change my password?",
"Do you sell gift cards?",
"I was charged twice for one order",
]
LABELS = ["billing", "shipping", "account"]
PROMPT = "Classify this support ticket. Reply with one word: billing, shipping, account or other.\n\nTicket: "Asking a model group and reading the headers
ask sends one ticket to a model group. with_raw_response is the OpenAI client's way to reach the HTTP response behind a call, and .parse() gives the usual reply object. The gateway's headers say which group answered, whether the answer came from the cache and what the call cost. max_tokens=300 leaves room for the model's reasoning, because a small budget on a gpt-oss model can return an empty answer.
def ask(group, text):
raw = client.chat.completions.with_raw_response.create(
model=group,
messages=[{"role": "user", "content": PROMPT + text}],
max_tokens=300,
)
h = raw.headers
label = (raw.parse().choices[0].message.content or "").strip().lower().strip(".")
cached = "x-litellm-cache-key" in h
cost = 0.0 if cached else float(h["x-litellm-response-cost"])
return label, h["x-litellm-model-group"], cached, costA cached answer is counted as zero here. The gateway still sends a cost header on a cached answer, so the app decides that a repeat costs nothing.
Escalating when quick cannot decide
triage asks quick first. If its word is not one of the three labels, the app asks careful and adds the two costs. The app does nothing about a failing quick, since the gateway's fallback handles that.
def triage(text):
label, served, cached, cost = ask("quick", text)
if label not in LABELS:
label, served, cached, extra = ask("careful", text)
cost += extra
return label, served, cached, costRunning the queue and handling a refusal
The loop sends every ticket. A ticket the guardrail refuses makes the OpenAI client raise BadRequestError, because the gateway answered 400, and the loop prints the gateway's message from err.body and moves on.
total = 0.0
for text in TICKETS:
try:
label, served, cached, cost = triage(text)
except BadRequestError as err:
print(f"{text:34} refused: {err.body['message']}")
continue
total += cost
print(f"{text:34} {label:9} {'cache' if cached else served:8} {cost:.6f}")
print(f"spent {total:.6f}")Saving the whole file
The example saves the complete program as app.py, reads it back, and checks that Python can compile it. It does not run it, since the gateway is not up. Save the file exactly as shown, because the next lesson depends on it.
from pathlib import Path
Path("app.py").write_text(r"""from openai import BadRequestError, OpenAI
client = OpenAI(base_url="http://127.0.0.1:4400", api_key="sk-gateway-1234")
TICKETS = [
"I was charged twice for one order",
"My parcel has not arrived",
"How do I change my password?",
"Do you sell gift cards?",
"I was charged twice for one order",
]
LABELS = ["billing", "shipping", "account"]
PROMPT = "Classify this support ticket. Reply with one word: billing, shipping, account or other.\n\nTicket: "
def ask(group, text):
raw = client.chat.completions.with_raw_response.create(
model=group,
messages=[{"role": "user", "content": PROMPT + text}],
max_tokens=300,
)
h = raw.headers
label = (raw.parse().choices[0].message.content or "").strip().lower().strip(".")
cached = "x-litellm-cache-key" in h
cost = 0.0 if cached else float(h["x-litellm-response-cost"])
return label, h["x-litellm-model-group"], cached, cost
def triage(text):
label, served, cached, cost = ask("quick", text)
if label not in LABELS:
label, served, cached, extra = ask("careful", text)
cost += extra
return label, served, cached, cost
total = 0.0
for text in TICKETS:
try:
label, served, cached, cost = triage(text)
except BadRequestError as err:
print(f"{text:34} refused: {err.body['message']}")
continue
total += cost
print(f"{text:34} {label:9} {'cache' if cached else served:8} {cost:.6f}")
print(f"spent {total:.6f}")
""")
text = Path("app.py").read_text()
print("saved app.py:", len(text.splitlines()), "lines")
print("compiles:", compile(text, "app.py", "exec") is not None)saved app.py: 46 lines compiles: True
- The file is the pieces, joined. Nothing was added that the pieces did not show: the client, the tickets,
ask,triageand the loop. - It compiles. A typo in a copied line would show here before it showed in a run.
- The gateway holds everything else. The app contains no provider name, no retry and no cache. Those are in
config.yaml.
The app vs a direct provider call
| desk app through the gateway | A program calling Groq directly | |
|---|---|---|
| Provider key | None, only the gateway key | In the program or its environment |
| Retries and fallback | In config.yaml | Written in the program |
| Repeated tickets | Answered from the cache | Paid for each time |
| Refusing a password ticket | The gateway returns 400 | The program has to check |
Where this app shape fits
- A triage step before a human. A cheap model labels the queue and a person handles the labels.
- Any program already written for the OpenAI client. Changing
base_urlis the whole change. - A team with several apps. Each reads the same two group names, and the fallback policy lives in one file.
x-litellm-model-group is how the app knows which group answered. If the app assumed quick always answers, a fallback would make its report wrong. The app reports the group the gateway names.Related
- Previous: Project: the gateway a team runs
- Next: Project: running the desk on a morning of tickets
- Reference: Using the proxy with the OpenAI client in the LiteLLM docs
- Add a sixth ticket to
TICKETSand read the file back to count the lines. - Change the
PROMPTto ask for a label and a one-line reason, and think about howtriagewould have to change. - Print
h["x-litellm-model-name"]next to the group inaskand read which provider model answered.
This is what real progress feels like.