Streaming responses: reading an answer as it arrives

A long answer from a model can take many seconds. Streaming sends it a piece at a time, so a person sees words appear instead of waiting for the whole reply.

OpenAI's API, and the many that copy its shape, stream with server-sent events: text lines starting with data: , each followed by a blank line, ending with data: [DONE]. Add streaming to the stand-in. The imports gain json and StreamingResponse, the request model gains stream: bool = False, and a generator writes the events:

def stream_words(answer: str):
    for word in answer.split(" "):
        yield "data: " + json.dumps({"delta": word + " "}) + "\n\n"
    yield "data: [DONE]\n\n"

yield makes a generator: a function that hands back values one at a time, each time something asks for the next. The endpoint wraps it in a StreamingResponse, which sends each piece as soon as it is produced:

    if request.stream:
        return StreamingResponse(stream_words(answer), media_type="text/event-stream")

Reading the stream

import json

from fastapi.testclient import TestClient
from model_api import app

model = TestClient(app, headers={"Authorization": "Bearer sk-local-123"})
body = {"model": "local", "stream": True,
        "messages": [{"role": "user", "content": "How do I change my password?"}]}

with model.stream("POST", "/v1/chat/completions", json=body) as response:
    for line in response.iter_lines():
        if line.startswith("data: ") and line != "data: [DONE]":
            print(json.loads(line[6:])["delta"], end="", flush=True)
print()

client.stream(...) sends the request without waiting for the whole body. iter_lines gives each line as it arrives. line[6:] drops the data: prefix, and end="" prints the pieces on one line.

python stream.py

The output is the same sentence as before, built from pieces. In a terminal or a chat window those pieces appear one after another; the blank lines between events and the [DONE] marker are skipped by the if.