A stand-in model API

Model providers put their models behind HTTP APIs. Build one that answers in the same shape, and every lesson after this can call a model API without a key or a bill.

OpenAI's Chat Completions API takes a POST to /v1/chat/completions with a model name and a list of messages, and many other providers accept the same shape. Start model_api.py with the request:

Examplemodel_api.py, part 1

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()


class Message(BaseModel):
    role: str
    content: str


class ChatRequest(BaseModel):
    model: str
    messages: list[Message]

A ChatRequest is a model name and a list of Messages, each a role and some content: the conversation shape from the Python course, now checked by Pydantic.

Examplemodel_api.py, part 2

def decide(text: str) -> str:
    text = text.lower()
    if "charged" in text or "refund" in text:
        return '{"category": "billing", "priority": 4}'
    if "parcel" in text or "arrived" in text:
        return '{"category": "shipping", "priority": 3}'
    return "I am not sure how to sort this one."

decide is the stand-in from Python for AI: JSON text for tickets it can sort, a sentence for one it cannot.

Examplemodel_api.py, part 3

@app.post("/v1/chat/completions")
def chat(request: ChatRequest):
    question = request.messages[-1].content
    answer = decide(question)
    return {
        "model": request.model,
        "choices": [{"index": 0, "message": {"role": "assistant", "content": answer}}],
        "usage": {"prompt_tokens": len(question.split()), "completion_tokens": len(answer.split())},
    }

The endpoint answers the last message. The reply sits inside choices, a list because some APIs can return several answers, and usage reports how much text went in and out. Real APIs count tokens there; this one counts words.

Calling it

Example

from fastapi.testclient import TestClient
from model_api import app

model = TestClient(app)
body = {"model": "local", "messages": [{"role": "user", "content": "My parcel has not arrived"}]}

response = model.post("/v1/chat/completions", json=body)
print(response.status_code)
print(response.json())

Example

data = response.json()
print(data["choices"][0]["message"]["content"])
print(data["usage"])

["choices"][0]["message"]["content"] is the path to the answer text in this shape, and you will write it many times. The answer is itself JSON text, which the service in part 4 checks with Pydantic.

Try it yourself

Send the password ticket and print the content.
Send a body with no messages and read the 422.
Add a system message before the user message. Does the answer change? Why not?

Previousuvicorn: running the API on a real server

This is what real progress feels like.