The triage service: an endpoint that calls a model

Everything so far meets in one endpoint. It takes a ticket, asks the model API, checks the answer, and replies with a status code that says what happened.

Exampleservice.py, part 1

import os
from typing import Literal

import httpx2
from fastapi import Depends, FastAPI, HTTPException
from pydantic import BaseModel, Field, ValidationError

app = FastAPI()


class Ticket(BaseModel):
    id: int
    text: str

Exampleservice.py, part 2

class Triage(BaseModel):
    category: Literal["billing", "shipping", "other"]
    priority: int = Field(ge=1, le=5)

A Ticket comes in as the body (lesson 6). Triage is the check on the model's answer, the same model as in Python for AI.

Exampleservice.py, part 3

def get_model_client():
    return httpx2.Client(
        base_url=os.environ.get("MODEL_API_URL", "http://127.0.0.1:8101"),
        headers={"Authorization": f"Bearer {os.environ['MODEL_API_KEY']}"},
        timeout=10.0,
    )

The model API's address and key come from environment variables (lesson 10), with a default address for your own machine, and a timeout suited to a model call (lesson 12).

Exampleservice.py, part 4

@app.post("/triage")
def triage(ticket: Ticket, model=Depends(get_model_client)):
    body = {"model": "local", "messages": [{"role": "user", "content": ticket.text}]}
    response = model.post("/v1/chat/completions", json=body)
    response.raise_for_status()
    answer = response.json()["choices"][0]["message"]["content"]
    try:
        result = Triage.model_validate_json(answer)
    except ValidationError:
        raise HTTPException(status_code=502, detail="the model gave an answer that failed the check")
    return {"id": ticket.id, **result.model_dump()}

model=Depends(get_model_client) asks FastAPI to call get_model_client for each request and pass in what it returns. This is dependency injection: the endpoint says what it needs, and something else provides it. Lesson 16 uses that to swap the model client in tests.

A failed check becomes 502: the service is fine, but a server it depends on gave a bad answer. The client gets a message instead of a wrong category.

Two servers

The service needs the model API running. Start both, the model API on 8101 and the service on 8100, then send two tickets:

Example

export MODEL_API_KEY=sk-local-123
uvicorn model_api:app --port 8101 --log-level warning &
uvicorn service:app --port 8100 --log-level warning &
sleep 3
curl -s -X POST localhost:8100/triage -H "Content-Type: application/json" \
  -d '{"id": 1, "text": "I was charged twice"}'
echo
curl -s -X POST localhost:8100/triage -H "Content-Type: application/json" \
  -d '{"id": 4, "text": "How do I change my password?"}'
echo
kill %1 %2

Ticket 1 went client, service, model API, and back, with the answer checked on the way. Ticket 4's answer failed the check, and the client was told so plainly.

Try it yourself

Stop only the model API, kill %1, and send a ticket. What status does the service return, and which lesson's error caused it?
Send four tickets in a row. The model API's rate limit from lesson 11 is still on; what happens to the third?
Wrap the post in the ask retry function from lesson 11.

PreviousStreaming responses: reading an answer as it arrives

Every expert started right here.