Testing the service with pytest

Every status code the service can send is a promise to its clients. Four tests check four of them, without starting a server or calling a model.

Exampletest_service.py, part 1

import pytest
from fastapi.testclient import TestClient

import model_api
import service

KEY = {"X-API-Key": "team-key-1"}


@pytest.fixture
def client():
    def fake_model_client():
        return TestClient(model_api.app, headers={"Authorization": "Bearer sk-local-123"})

    service.app.dependency_overrides[service.get_model_client] = fake_model_client
    yield TestClient(service.app)
    service.app.dependency_overrides.clear()

A fixture is a function pytest runs to prepare something a test needs. A test that has a parameter called client receives what this fixture yields. The code after yield runs when the test is finished, so each test starts with no overrides left behind.

Exampletest_service.py, part 2

def test_billing_ticket(client):
    response = client.post("/triage", json={"id": 1, "text": "I was charged twice"}, headers=KEY)
    assert response.status_code == 200
    assert response.json() == {"id": 1, "category": "billing", "priority": 4}


def test_unclear_answer_is_502(client):
    response = client.post("/triage", json={"id": 4, "text": "How do I change my password?"}, headers=KEY)
    assert response.status_code == 502


def test_missing_key_is_401(client):
    response = client.post("/triage", json={"id": 1, "text": "I was charged twice"})
    assert response.status_code == 401


def test_missing_text_is_422(client):
    response = client.post("/triage", json={"id": 1}, headers=KEY)
    assert response.status_code == 422

One test per outcome: a sorted ticket, an answer that failed the check, a missing key, a missing field. Comparing the whole JSON in the first test also catches a field that appears or disappears.

One more file. The Starlette version installed today prints a deprecation warning from inside its own test client on every run; it is about Starlette's code, not yours, so tell pytest to leave that one out:

Examplepytest.ini

[pytest]
filterwarnings =
    ignore:The anyio.abc.BlockingPortal alias is deprecated:DeprecationWarning

Example

MODEL_API_KEY=unused pytest -q

Four passed. MODEL_API_KEY=unused in front of the command sets the variable for that one command. It is needed only because get_model_client reads it, and the override means that function never runs.

A test that catches a real mistake

Suppose someone makes the service friendlier by returning a guess when the check fails, {"id": ticket.id, "category": "other", "priority": 1}, instead of raising 502:

Exampleservice.py, the except block changed

    except ValidationError:
        return {"id": ticket.id, "category": "other", "priority": 1}

Example

MODEL_API_KEY=unused pytest -q

The test for the unclear ticket fails and names it: the service answered 200 with a guessed category, which a client would store as a real result.

Try it yourself

Add a test that a wrong X-API-Key gets 401.
Add a test for the rate limit: post three tickets and assert the third is not 200. What status does the service send, and should it?
Parametrize test_billing_ticket over two billing tickets, as in Python for AI.

PreviousDepends: protecting and swapping parts of a service

You understood something today that you didn't yesterday.