Tests for what the desk sends

The desk's observation names, its masking and its warnings are an interface other things depend on. Tests keep them from changing by accident.

The fixture sets the desk up the way app.py does, masking included, once for the file.

Exampletest_support_desk.py

import json

import pytest
from langfuse import Langfuse
from langfuse.openai import OpenAI

import local_langfuse
import support_desk
from privacy import mask_cards
from shop_model import reply

Exampletest_support_desk.py, continued

@pytest.fixture(scope="module")
def langfuse():
    url = local_langfuse.start(model=reply)
    client = Langfuse(public_key="pk-lf-local", secret_key="sk-lf-local", base_url=url, mask_otel_spans=mask_cards)
    support_desk.model = OpenAI(base_url=f"{url}/v1", api_key="not-a-real-key")
    client.create_prompt(name="desk-system", prompt=support_desk.FALLBACK, labels=["production"])
    return client

Each test starts from an empty server, so one test cannot see another's observations.

Exampletest_support_desk.py, continued

def spans_for(langfuse, number, ticket):
    local_langfuse.SPANS.clear()
    support_desk.handle(number, ticket, customer="cust-1", chat="chat-1")
    langfuse.flush()
    return local_langfuse.SPANS

The first test pins the shape of a trace: the names, in order, that filters and dashboards rely on.

Exampletest_support_desk.py, continued

def test_trace_shape(langfuse):
    names = [span["name"] for span in spans_for(langfuse, "t-1", "Where is my order A17?")]
    assert names == ["lookup-order", "write-reply", "answer-ticket", "support-ticket"]


def test_card_numbers_never_leave(langfuse):
    spans = spans_for(langfuse, "t-2", "Card 4111 1111 1111 1111 was charged twice on A17")
    assert spans and "4111" not in json.dumps([span["attributes"] for span in spans])

The second searches every exported attribute for the card number, which is the only way to be sure masking covered the model call as well as the ticket.

Exampletest_support_desk.py, continued

def test_feedback_lands_on_the_ticket_trace(langfuse):
    trace = spans_for(langfuse, "t-3", "Is B22 on its way?")[0]["trace"]
    support_desk.record_feedback("t-3", helpful=False)
    langfuse.flush()
    assert local_langfuse.SCORES[-1]["traceId"] == trace

Example

pytest -q -p no:warnings test_support_desk.py

Both pass against the stand-in server, with no key and no network, so they can run on every change.

Try it yourself

Rename an observation in the desk and watch the first test fail.
Add a test that a missing order produces a warning.
Add a test that the customer's id reaches every observation.

PreviousThree tickets, and the card number

This is what real progress feels like.