Datasets: the tickets the desk must get right
A dataset is a named list of inputs with expected outputs, stored in Langfuse. It is the fixed set of cases every version of the desk is tested against.
Scores from lessons 23 and 24 judge traffic that already happened. Before a new prompt reaches customers, you want to run it on known cases and compare. Those cases live in a dataset: each item has an input, an optional expected output, and optional metadata.
CASES = [
("Where is my order A17?", "shipped on 3 March"),
("Is B22 on its way?", "could not find order B22"),
("I want a refund for A17", "need approval"),
]
langfuse.create_dataset(name="desk-tickets", description="Tickets the desk must answer correctly")
for ticket, expected in CASES:
langfuse.create_dataset_item(dataset_name="desk-tickets", input=ticket, expected_output=expected)The expected outputs are phrases a correct answer must contain, not whole answers; the next lesson's check looks for them. A dataset name is unique in a project, and a slash in it, like desk/tickets, shows as a folder in Langfuse's interface.
dataset = langfuse.get_dataset("desk-tickets")
for item in dataset.items:
print(item.input, "->", item.expected_output)
print(local_langfuse.REQUESTS)View the code here
from langfuse import Langfuse
import local_langfuse
url = local_langfuse.start()
langfuse = Langfuse(public_key="pk-lf-local", secret_key="sk-lf-local", base_url=url)
CASES = [
("Where is my order A17?", "shipped on 3 March"),
("Is B22 on its way?", "could not find order B22"),
("I want a refund for A17", "need approval"),
]
langfuse.create_dataset(name="desk-tickets", description="Tickets the desk must answer correctly")
for ticket, expected in CASES:
langfuse.create_dataset_item(dataset_name="desk-tickets", input=ticket, expected_output=expected)
langfuse.create_dataset_item(
dataset_name="desk-tickets",
input="Hello, where is my parcel?",
expected_output="order number",
source_trace_id=langfuse.create_trace_id(seed="ticket-1044"),
)
print(langfuse.get_dataset("desk-tickets").items[-1].source_trace_id)
"""A stand-in for the Langfuse server, on your own machine.
The Langfuse SDK sends observations to /api/public/otel/v1/traces and uses a
REST API for scores, prompts and datasets. This server answers those requests
on 127.0.0.1 and keeps what it receives in lists you can print. Given a model
function, it also answers like OpenAI's chat API. start() blocks every
connection to another computer for the rest of the program.
"""
import json
import socket
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, unquote, urlparse
from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest
SPANS, SCORES, REQUESTS = [], [], [] # observations, scores, "METHOD /path" of each request
PROMPTS, DATASETS = {}, {} # prompt name -> its versions, dataset name -> dataset
MODEL = None # answers /v1/chat/completions when set
NOW = "2026-03-03T09:00:00Z"
def _value(v):
kind = v.WhichOneof("value")
if kind == "array_value":
return [_value(item) for item in v.array_value.values]
return getattr(v, kind) if kind else None
def _spans(body):
request = ExportTraceServiceRequest()
request.ParseFromString(body)
for resource in request.resource_spans:
for scope in resource.scope_spans:
for span in scope.spans:
SPANS.append({
"name": span.name, "id": span.span_id.hex(), "trace": span.trace_id.hex(),
"parent": span.parent_span_id.hex(), "scope": scope.scope.name,
"attributes": {a.key: _value(a.value) for a in span.attributes},
})
return {}
def _scores(batch):
for event in batch:
if event["type"] == "score-create":
SCORES.append(event["body"])
return {"successes": [{"id": e["id"], "status": 201} for e in batch], "errors": []}
def _prompt(name, query, body):
versions = PROMPTS.setdefault(name, [])
if body is not None: # a new version: it takes "latest" and any labels it asks for
labels = body.get("labels", []) + ["latest"]
for old in versions:
old["labels"] = [label for label in old["labels"] if label not in labels]
versions.append({"name": name, "type": body.get("type", "text"), "prompt": body["prompt"],
"config": body.get("config") or {}, "labels": labels, "tags": [],
"version": len(versions) + 1})
return versions[-1]
if "version" in query:
return versions[int(query["version"]) - 1]
label = query.get("label", "production")
return next(v for v in versions if label in v["labels"])
def _relabel(path, body):
name, _, version = path.split("/")[-3:]
versions = PROMPTS[unquote(name)]
for old in versions:
old["labels"] = [label for label in old["labels"] if label not in body["newLabels"]]
versions[int(version) - 1]["labels"] += body["newLabels"]
return versions[int(version) - 1]
def _dataset(path, query, body):
if path == "/api/public/v2/datasets":
DATASETS[body["name"]] = {"id": body["name"], "name": body["name"], "items": [],
"metadata": body.get("metadata"), "projectId": "local",
"createdAt": NOW, "updatedAt": NOW}
return DATASETS[body["name"]]
if path.startswith("/api/public/v2/datasets/"):
return DATASETS[unquote(path.rsplit("/", 1)[1])]
if path == "/api/public/dataset-items" and body is not None:
dataset = DATASETS[body["datasetName"]]
item = {"id": body.get("id") or f"item-{len(dataset['items']) + 1}", "status": "ACTIVE",
"input": body.get("input"), "expectedOutput": body.get("expectedOutput"),
"metadata": body.get("metadata"), "sourceTraceId": body.get("sourceTraceId"),
"datasetId": dataset["id"], "datasetName": dataset["name"],
"createdAt": NOW, "updatedAt": NOW, "mediaReferences": []}
dataset["items"].append(item)
return item
if path == "/api/public/dataset-items":
items = DATASETS[query["datasetName"]]["items"]
return {"data": items, "meta": {"page": 1, "limit": 50, "totalItems": len(items), "totalPages": 1}}
return {"id": "run-item", "datasetRunId": body["runName"], "datasetRunName": body["runName"],
"datasetItemId": body["datasetItemId"], "traceId": body["traceId"],
"observationId": body.get("observationId"), "createdAt": NOW, "updatedAt": NOW}
def _chat(body):
text, usage = MODEL(body["messages"])
return {"id": "chatcmpl-1", "object": "chat.completion", "created": 0, "model": body["model"],
"choices": [{"index": 0, "finish_reason": "stop", "message": {"role": "assistant", "content": text}}],
"usage": {"prompt_tokens": usage["input"], "completion_tokens": usage["output"],
"total_tokens": usage["input"] + usage["output"]}}
class Handler(BaseHTTPRequestHandler):
def _handle(self):
url = urlparse(self.path)
query = {key: values[0] for key, values in parse_qs(url.query).items()}
raw = self.rfile.read(int(self.headers.get("Content-Length") or 0))
body = json.loads(raw) if raw and "json" in self.headers.get("Content-Type", "") else None
REQUESTS.append(f"{self.command} {url.path}")
try:
if url.path.endswith("/otel/v1/traces"):
return self._reply(200, _spans(raw))
if url.path == "/api/public/ingestion":
return self._reply(207, _scores(body["batch"]))
if url.path == "/api/public/projects":
return self._reply(200, {"data": [{"id": "local", "name": "shop-desk", "metadata": {},
"organization": {"id": "local", "name": "local"}}]})
if url.path.startswith("/api/public/v2/prompts/") and self.command == "PATCH":
return self._reply(200, _relabel(url.path, body))
if url.path.startswith("/api/public/v2/prompts"):
name = body["name"] if body else unquote(url.path.rsplit("/", 1)[1])
return self._reply(200, _prompt(name, query, body))
if "dataset" in url.path:
return self._reply(200, _dataset(url.path, query, body))
if url.path.endswith("/chat/completions") and MODEL:
return self._reply(200, _chat(body))
except (KeyError, StopIteration):
pass
self._reply(404, {"message": f"not found: {url.path}"})
do_GET = do_POST = do_PATCH = _handle
def _reply(self, code, data):
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.end_headers()
self.wfile.write(json.dumps(data).encode())
def log_message(self, *args):
pass
def _local_only():
"""Refuse to look up or connect to any other computer, so nothing leaves by mistake."""
lookup, connect = socket.getaddrinfo, socket.socket.connect
def guarded_lookup(host, *args, **kwargs):
if host not in ("127.0.0.1", "localhost", "::1"):
raise ConnectionRefusedError(f"blocked a connection to {host}: this lab stays on your machine")
return lookup(host, *args, **kwargs)
def guarded_connect(sock, address):
if isinstance(address, tuple) and address[0] not in ("127.0.0.1", "::1"):
raise ConnectionRefusedError(f"blocked a connection to {address[0]}: this lab stays on your machine")
return connect(sock, address)
socket.getaddrinfo, socket.socket.connect = guarded_lookup, guarded_connect
def start(model=None):
"""Start the server in the background and return its address."""
global MODEL
_local_only()
MODEL = model
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
threading.Thread(target=server.serve_forever, daemon=True).start()
return f"http://127.0.0.1:{server.server_address[1]}"
def tree(*fields):
"""Print the observations received so far, each under its parent, with any fields asked for."""
ids = {span["id"] for span in SPANS}
def walk(span, depth):
attributes = span["attributes"]
line = " " * depth + f"{span['name']} ({attributes.get('langfuse.observation.type', 'span')})"
for field in fields:
line += f" {field}={attributes.get('langfuse.observation.' + field)!r}"
print(line)
for child in SPANS:
if child["parent"] == span["id"]:
walk(child, depth + 1)
for root in [span for span in SPANS if span["parent"] not in ids]:
walk(root, 0)
python dataset.pyWhere is my order A17? -> shipped on 3 March Is B22 on its way? -> could not find order B22 I want a refund for A17 -> need approval ['POST /api/public/v2/datasets', 'POST /api/public/dataset-items', 'POST /api/public/dataset-items', 'POST /api/public/dataset-items', 'GET /api/public/v2/datasets/desk-tickets', 'GET /api/public/dataset-items']
get_dataset fetched the dataset, then its items page by page. Each item also has an id, which the SDK generated when it created the item, and the name of its dataset.
A case from a real ticket
The best new cases are tickets the desk got wrong. source_trace_id records which trace an item came from, so you can go back to what happened.
langfuse.create_dataset_item(
dataset_name="desk-tickets",
input="Hello, where is my parcel?",
expected_output="order number",
source_trace_id=langfuse.create_trace_id(seed="ticket-1044"),
)
print(langfuse.get_dataset("desk-tickets").items[-1].source_trace_id)python dataset.py3d9b1d006e95cb2fb371b653a9b2c4ce
The item points at ticket 1044's trace, found again by its seeded id. In Langfuse's interface the same thing is a button on any observation. Changing an item, adding one or archiving one creates a new version of the dataset, and get_dataset(..., version=...) fetches it as it was at a given time, so an old test run can be repeated exactly.
- Add
metadata={"topic": "refunds"}to the refund item and print it. - Name the dataset
desk/ticketsand fetch it by the same name. - Make the input a dictionary,
{"ticket": ..., "customer": ...}, and print the items.
Every expert started right here.