Datasets: the tickets the desk must get right

A dataset is a named list of inputs with expected outputs, stored in Langfuse. It is the fixed set of cases every version of the desk is tested against.

Scores from lessons 23 and 24 judge traffic that already happened. Before a new prompt reaches customers, you want to run it on known cases and compare. Those cases live in a dataset: each item has an input, an optional expected output, and optional metadata.

CASES = [
    ("Where is my order A17?", "shipped on 3 March"),
    ("Is B22 on its way?", "could not find order B22"),
    ("I want a refund for A17", "need approval"),
]

langfuse.create_dataset(name="desk-tickets", description="Tickets the desk must answer correctly")
for ticket, expected in CASES:
    langfuse.create_dataset_item(dataset_name="desk-tickets", input=ticket, expected_output=expected)

The expected outputs are phrases a correct answer must contain, not whole answers; the next lesson's check looks for them. A dataset name is unique in a project, and a slash in it, like desk/tickets, shows as a folder in Langfuse's interface.

dataset = langfuse.get_dataset("desk-tickets")
for item in dataset.items:
    print(item.input, "->", item.expected_output)
print(local_langfuse.REQUESTS)

python dataset.py

get_dataset fetched the dataset, then its items page by page. Each item also has an id, which the SDK generated when it created the item, and the name of its dataset.

A case from a real ticket

The best new cases are tickets the desk got wrong. source_trace_id records which trace an item came from, so you can go back to what happened.

langfuse.create_dataset_item(
    dataset_name="desk-tickets",
    input="Hello, where is my parcel?",
    expected_output="order number",
    source_trace_id=langfuse.create_trace_id(seed="ticket-1044"),
)
print(langfuse.get_dataset("desk-tickets").items[-1].source_trace_id)

python dataset.py

The item points at ticket 1044's trace, found again by its seeded id. In Langfuse's interface the same thing is a button on any observation. Changing an item, adding one or archiving one creates a new version of the dataset, and get_dataset(..., version=...) fetches it as it was at a given time, so an old test run can be repeated exactly.