LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

Project: the desk that calls it

The desk app is a short Python program that sends support tickets to the gateway with the OpenAI client and sorts each one into billing, shipping or account.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

Project: the gateway a team runs started the server. The app is its client, and it holds none of the gateway's work: no provider key, no retry loop, no cache and no model names beyond the two groups. This lesson builds the file, app.py, in pieces and then saves it. This is the one version of app.py in the project. Project: running the desk on a morning of tickets runs it unchanged, and if a change is ever needed, that change is made to this file and stated.

Connecting to the gateway

The OpenAI client works against the gateway because the gateway speaks the same API. Only base_url and the key change: base_url points at the gateway's address and api_key is the master key from the config.

python
from openai import BadRequestError, OpenAI

client = OpenAI(base_url="http://127.0.0.1:4400", api_key="sk-gateway-1234")

Listing the tickets and the question

Five tickets stand for a morning's queue. One mentions a password, which the guardrail will refuse. One does not fit the three labels. The first ticket comes back at the end, as repeats do on a real desk. The prompt asks for one word.

python
TICKETS = [
    "I was charged twice for one order",
    "My parcel has not arrived",
    "How do I change my password?",
    "Do you sell gift cards?",
    "I was charged twice for one order",
]
LABELS = ["billing", "shipping", "account"]
PROMPT = "Classify this support ticket. Reply with one word: billing, shipping, account or other.\n\nTicket: "

Asking a model group and reading the headers

ask sends one ticket to a model group. with_raw_response is the OpenAI client's way to reach the HTTP response behind a call, and .parse() gives the usual reply object. The gateway's headers say which group answered, whether the answer came from the cache and what the call cost. max_tokens=300 leaves room for the model's reasoning, because a small budget on a gpt-oss model can return an empty answer.

python
def ask(group, text):
    raw = client.chat.completions.with_raw_response.create(
        model=group,
        messages=[{"role": "user", "content": PROMPT + text}],
        max_tokens=300,
    )
    h = raw.headers
    label = (raw.parse().choices[0].message.content or "").strip().lower().strip(".")
    cached = "x-litellm-cache-key" in h
    cost = 0.0 if cached else float(h["x-litellm-response-cost"])
    return label, h["x-litellm-model-group"], cached, cost

A cached answer is counted as zero here. The gateway still sends a cost header on a cached answer, so the app decides that a repeat costs nothing.

Escalating when quick cannot decide

triage asks quick first. If its word is not one of the three labels, the app asks careful and adds the two costs. The app does nothing about a failing quick, since the gateway's fallback handles that.

python
def triage(text):
    label, served, cached, cost = ask("quick", text)
    if label not in LABELS:
        label, served, cached, extra = ask("careful", text)
        cost += extra
    return label, served, cached, cost

Running the queue and handling a refusal

The loop sends every ticket. A ticket the guardrail refuses makes the OpenAI client raise BadRequestError, because the gateway answered 400, and the loop prints the gateway's message from err.body and moves on.

python
total = 0.0
for text in TICKETS:
    try:
        label, served, cached, cost = triage(text)
    except BadRequestError as err:
        print(f"{text:34} refused: {err.body['message']}")
        continue
    total += cost
    print(f"{text:34} {label:9} {'cache' if cached else served:8} {cost:.6f}")
print(f"spent {total:.6f}")

Saving the whole file

The example saves the complete program as app.py, reads it back, and checks that Python can compile it. It does not run it, since the gateway is not up. Save the file exactly as shown, because the next lesson depends on it.

ExampleSaving app.py, the only version of it
from pathlib import Path

Path("app.py").write_text(r"""from openai import BadRequestError, OpenAI

client = OpenAI(base_url="http://127.0.0.1:4400", api_key="sk-gateway-1234")

TICKETS = [
    "I was charged twice for one order",
    "My parcel has not arrived",
    "How do I change my password?",
    "Do you sell gift cards?",
    "I was charged twice for one order",
]
LABELS = ["billing", "shipping", "account"]
PROMPT = "Classify this support ticket. Reply with one word: billing, shipping, account or other.\n\nTicket: "


def ask(group, text):
    raw = client.chat.completions.with_raw_response.create(
        model=group,
        messages=[{"role": "user", "content": PROMPT + text}],
        max_tokens=300,
    )
    h = raw.headers
    label = (raw.parse().choices[0].message.content or "").strip().lower().strip(".")
    cached = "x-litellm-cache-key" in h
    cost = 0.0 if cached else float(h["x-litellm-response-cost"])
    return label, h["x-litellm-model-group"], cached, cost


def triage(text):
    label, served, cached, cost = ask("quick", text)
    if label not in LABELS:
        label, served, cached, extra = ask("careful", text)
        cost += extra
    return label, served, cached, cost


total = 0.0
for text in TICKETS:
    try:
        label, served, cached, cost = triage(text)
    except BadRequestError as err:
        print(f"{text:34} refused: {err.body['message']}")
        continue
    total += cost
    print(f"{text:34} {label:9} {'cache' if cached else served:8} {cost:.6f}")
print(f"spent {total:.6f}")
""")

text = Path("app.py").read_text()
print("saved app.py:", len(text.splitlines()), "lines")
print("compiles:", compile(text, "app.py", "exec") is not None)
  • The file is the pieces, joined. Nothing was added that the pieces did not show: the client, the tickets, ask, triage and the loop.
  • It compiles. A typo in a copied line would show here before it showed in a run.
  • The gateway holds everything else. The app contains no provider name, no retry and no cache. Those are in config.yaml.

The app vs a direct provider call

desk app through the gatewayA program calling Groq directly
Provider keyNone, only the gateway keyIn the program or its environment
Retries and fallbackIn config.yamlWritten in the program
Repeated ticketsAnswered from the cachePaid for each time
Refusing a password ticketThe gateway returns 400The program has to check

Where this app shape fits

  • A triage step before a human. A cheap model labels the queue and a person handles the labels.
  • Any program already written for the OpenAI client. Changing base_url is the whole change.
  • A team with several apps. Each reads the same two group names, and the fallback policy lives in one file.
Watch out. Reading x-litellm-model-group is how the app knows which group answered. If the app assumed quick always answers, a fallback would make its report wrong. The app reports the group the gateway names.
Try it yourself
  • Add a sixth ticket to TICKETS and read the file back to count the lines.
  • Change the PROMPT to ask for a label and a one-line reason, and think about how triage would have to change.
  • Print h["x-litellm-model-name"] next to the group in ask and read which provider model answered.

This is what real progress feels like.