LangfuseLangfuse Python SDK 4.15.4 · Python 3.10+
Dashboard
0%

Comparing prompts on a Langfuse dataset

dataset.run_experiment links every run to the dataset, and a run evaluator scores the run as a whole. Two runs with two prompts give two numbers to compare.

Lesson 26 ran on a list in your code. Run on a dataset stored in Langfuse, each run is recorded against the dataset, so runs made weeks apart, by different people, can be compared side by side. This lesson compares two candidate system prompts.

A model that follows instructions

A prompt comparison needs a model that reads its prompt. The stand-in from lesson 6 ignores the system message, so it learns two instructions: to thank the customer, and to never share order details.

Exampleshop_model.py, the new lines in reply
    rules = messages[0]["content"].lower() if messages[0]["role"] == "system" else ""
    ...
    elif lookup and "never share" in rules:
        text = "I cannot share order details."
    ...
    if "thank" in rules:
        text = "Thanks for waiting. " + text

The full file is lesson 6's with these lines added: rules is the lowercased system message, the new elif sits before the not-found case, and the thank-you check runs right before the usage count. Save it; lessons from here on use this version.

Two runs on one dataset

Examplecompare.py, after the setup lines, from desk import answer, lesson 25's dataset lines and lesson 26's fixed evaluator
def average(*, item_results, **kwargs):
    values = [e.value for row in item_results for e in row.evaluations]
    return Evaluation(name="average", value=sum(values) / len(values))


def compare(system, run_name):
    def task(*, item, **kwargs):
        return answer(item.input, system=system)

    result = dataset.run_experiment(name="desk-prompts", run_name=run_name, task=task,
                                    evaluators=[contains_expected], run_evaluators=[average])
    print(run_name, "average:", round(result.run_evaluations[0].value, 2))

A run evaluator receives every item's result and returns one evaluation for the whole run, here the average. The task gets a dataset item, so the ticket is item.input. run_name names each run; without one the SDK makes one from the name and the time.

Examplecompare.py, continued
dataset = langfuse.get_dataset("desk-tickets")
compare("Answer in one short sentence. Thank the customer.", "thank-the-customer")
compare("Answer in one short sentence. Never share order details.", "never-share")

langfuse.flush()
print([r for r in local_langfuse.REQUESTS if "run-items" in r])
print([(s["name"], s.get("datasetRunId"), s["value"]) for s in local_langfuse.SCORES if s.get("datasetRunId")])
Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports these files. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep them in the same folder.
View the code here
desk.py
"""The shop's support desk, traced: the pieces from lessons 4 to 8 in one module."""
import re

from langfuse import get_client, observe

from shop_model import reply

ORDERS = {"A17": "shipped on 3 March"}
SYSTEM = "You answer customers of a small online shop in one short sentence."


@observe(name="lookup-order", as_type="tool")
def lookup_order(order_id):
    status = ORDERS.get(order_id, "not found")
    if status == "not found":
        get_client().update_current_span(level="WARNING", status_message=f"no order {order_id}")
    return status


@observe(name="write-reply", as_type="generation")
def ask_model(messages):
    text, usage = reply(messages)
    get_client().update_current_generation(model="shop-model", usage_details=usage)
    return text


@observe(name="answer-ticket", as_type="agent")
def answer(ticket, system=SYSTEM):
    order = re.search(r"[A-Z]\d{2}", ticket)
    facts = f"\nLookup: {order.group()} {lookup_order(order.group())}" if order else ""
    return ask_model([{"role": "system", "content": system}, {"role": "user", "content": ticket + facts}])
compare.py
from langfuse import Langfuse

import local_langfuse

url = local_langfuse.start()
langfuse = Langfuse(public_key="pk-lf-local", secret_key="sk-lf-local", base_url=url)

from desk import answer

from langfuse import Evaluation

CASES = [
    ("Where is my order A17?", "shipped on 3 March"),
    ("Is B22 on its way?", "could not find order B22"),
    ("I want a refund for A17", "need approval"),
]

langfuse.create_dataset(name="desk-tickets", description="Tickets the desk must answer correctly")
for ticket, expected in CASES:
    langfuse.create_dataset_item(dataset_name="desk-tickets", input=ticket, expected_output=expected)


def contains_expected(*, output, expected_output, **kwargs):
    if expected_output is None:
        return Evaluation(name="contains-expected", value=0.0, comment="no expected output to compare")
    found = expected_output.lower() in output.lower()
    return Evaluation(name="contains-expected", value=1.0 if found else 0.0)


def average(*, item_results, **kwargs):
    values = [e.value for row in item_results for e in row.evaluations]
    return Evaluation(name="average", value=sum(values) / len(values))


def compare(system, run_name):
    def task(*, item, **kwargs):
        return answer(item.input, system=system)

    result = dataset.run_experiment(name="desk-prompts", run_name=run_name, task=task,
                                    evaluators=[contains_expected], run_evaluators=[average])
    print(run_name, "average:", round(result.run_evaluations[0].value, 2))

dataset = langfuse.get_dataset("desk-tickets")
compare("Answer in one short sentence. Thank the customer.", "thank-the-customer")
compare("Answer in one short sentence. Never share order details.", "never-share")

langfuse.flush()
print([r for r in local_langfuse.REQUESTS if "run-items" in r])
print([(s["name"], s.get("datasetRunId"), s["value"]) for s in local_langfuse.SCORES if s.get("datasetRunId")])
shop_model.py
"""A stand-in chat model for the shop's support desk: no key, no network."""
import re


def reply(messages):
    """Answer the last message, and count words as tokens the way a chat model reports usage."""
    rules = messages[0]["content"].lower() if messages[0]["role"] == "system" else ""
    last = messages[-1]["content"]
    lookup = re.search(r"Lookup: (\S+) (.+)", last)
    if "refund" in last.lower():
        text = "Refunds need approval, so a person will review your request."
    elif lookup and "never share" in rules:
        text = "I cannot share order details."
    elif lookup and lookup.group(2) == "not found":
        text = f"I could not find order {lookup.group(1)}. Please check the number."
    elif lookup:
        text = f"Order {lookup.group(1)} {lookup.group(2)}."
    else:
        text = "Could you send your order number?"
    if "thank" in rules:
        text = "Thanks for waiting. " + text
    used = sum(len(message["content"].split()) for message in messages)
    return text, {"input": used, "output": len(text.split())}
local_langfuse.py
"""A stand-in for the Langfuse server, on your own machine.

The Langfuse SDK sends observations to /api/public/otel/v1/traces and uses a
REST API for scores, prompts and datasets. This server answers those requests
on 127.0.0.1 and keeps what it receives in lists you can print. Given a model
function, it also answers like OpenAI's chat API. start() blocks every
connection to another computer for the rest of the program.
"""
import json
import socket
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, unquote, urlparse

from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest

SPANS, SCORES, REQUESTS = [], [], []  # observations, scores, "METHOD /path" of each request
PROMPTS, DATASETS = {}, {}             # prompt name -> its versions, dataset name -> dataset
MODEL = None                           # answers /v1/chat/completions when set
NOW = "2026-03-03T09:00:00Z"


def _value(v):
    kind = v.WhichOneof("value")
    if kind == "array_value":
        return [_value(item) for item in v.array_value.values]
    return getattr(v, kind) if kind else None


def _spans(body):
    request = ExportTraceServiceRequest()
    request.ParseFromString(body)
    for resource in request.resource_spans:
        for scope in resource.scope_spans:
            for span in scope.spans:
                SPANS.append({
                    "name": span.name, "id": span.span_id.hex(), "trace": span.trace_id.hex(),
                    "parent": span.parent_span_id.hex(), "scope": scope.scope.name,
                    "attributes": {a.key: _value(a.value) for a in span.attributes},
                })
    return {}


def _scores(batch):
    for event in batch:
        if event["type"] == "score-create":
            SCORES.append(event["body"])
    return {"successes": [{"id": e["id"], "status": 201} for e in batch], "errors": []}


def _prompt(name, query, body):
    versions = PROMPTS.setdefault(name, [])
    if body is not None:  # a new version: it takes "latest" and any labels it asks for
        labels = body.get("labels", []) + ["latest"]
        for old in versions:
            old["labels"] = [label for label in old["labels"] if label not in labels]
        versions.append({"name": name, "type": body.get("type", "text"), "prompt": body["prompt"],
                         "config": body.get("config") or {}, "labels": labels, "tags": [],
                         "version": len(versions) + 1})
        return versions[-1]
    if "version" in query:
        return versions[int(query["version"]) - 1]
    label = query.get("label", "production")
    return next(v for v in versions if label in v["labels"])


def _relabel(path, body):
    name, _, version = path.split("/")[-3:]
    versions = PROMPTS[unquote(name)]
    for old in versions:
        old["labels"] = [label for label in old["labels"] if label not in body["newLabels"]]
    versions[int(version) - 1]["labels"] += body["newLabels"]
    return versions[int(version) - 1]


def _dataset(path, query, body):
    if path == "/api/public/v2/datasets":
        DATASETS[body["name"]] = {"id": body["name"], "name": body["name"], "items": [],
                                  "metadata": body.get("metadata"), "projectId": "local",
                                  "createdAt": NOW, "updatedAt": NOW}
        return DATASETS[body["name"]]
    if path.startswith("/api/public/v2/datasets/"):
        return DATASETS[unquote(path.rsplit("/", 1)[1])]
    if path == "/api/public/dataset-items" and body is not None:
        dataset = DATASETS[body["datasetName"]]
        item = {"id": body.get("id") or f"item-{len(dataset['items']) + 1}", "status": "ACTIVE",
                "input": body.get("input"), "expectedOutput": body.get("expectedOutput"),
                "metadata": body.get("metadata"), "sourceTraceId": body.get("sourceTraceId"),
                "datasetId": dataset["id"], "datasetName": dataset["name"],
                "createdAt": NOW, "updatedAt": NOW, "mediaReferences": []}
        dataset["items"].append(item)
        return item
    if path == "/api/public/dataset-items":
        items = DATASETS[query["datasetName"]]["items"]
        return {"data": items, "meta": {"page": 1, "limit": 50, "totalItems": len(items), "totalPages": 1}}
    return {"id": "run-item", "datasetRunId": body["runName"], "datasetRunName": body["runName"],
            "datasetItemId": body["datasetItemId"], "traceId": body["traceId"],
            "observationId": body.get("observationId"), "createdAt": NOW, "updatedAt": NOW}


def _chat(body):
    text, usage = MODEL(body["messages"])
    return {"id": "chatcmpl-1", "object": "chat.completion", "created": 0, "model": body["model"],
            "choices": [{"index": 0, "finish_reason": "stop", "message": {"role": "assistant", "content": text}}],
            "usage": {"prompt_tokens": usage["input"], "completion_tokens": usage["output"],
                      "total_tokens": usage["input"] + usage["output"]}}


class Handler(BaseHTTPRequestHandler):
    def _handle(self):
        url = urlparse(self.path)
        query = {key: values[0] for key, values in parse_qs(url.query).items()}
        raw = self.rfile.read(int(self.headers.get("Content-Length") or 0))
        body = json.loads(raw) if raw and "json" in self.headers.get("Content-Type", "") else None
        REQUESTS.append(f"{self.command} {url.path}")
        try:
            if url.path.endswith("/otel/v1/traces"):
                return self._reply(200, _spans(raw))
            if url.path == "/api/public/ingestion":
                return self._reply(207, _scores(body["batch"]))
            if url.path == "/api/public/projects":
                return self._reply(200, {"data": [{"id": "local", "name": "shop-desk", "metadata": {},
                                                   "organization": {"id": "local", "name": "local"}}]})
            if url.path.startswith("/api/public/v2/prompts/") and self.command == "PATCH":
                return self._reply(200, _relabel(url.path, body))
            if url.path.startswith("/api/public/v2/prompts"):
                name = body["name"] if body else unquote(url.path.rsplit("/", 1)[1])
                return self._reply(200, _prompt(name, query, body))
            if "dataset" in url.path:
                return self._reply(200, _dataset(url.path, query, body))
            if url.path.endswith("/chat/completions") and MODEL:
                return self._reply(200, _chat(body))
        except (KeyError, StopIteration):
            pass
        self._reply(404, {"message": f"not found: {url.path}"})

    do_GET = do_POST = do_PATCH = _handle

    def _reply(self, code, data):
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps(data).encode())

    def log_message(self, *args):
        pass


def _local_only():
    """Refuse to look up or connect to any other computer, so nothing leaves by mistake."""
    lookup, connect = socket.getaddrinfo, socket.socket.connect

    def guarded_lookup(host, *args, **kwargs):
        if host not in ("127.0.0.1", "localhost", "::1"):
            raise ConnectionRefusedError(f"blocked a connection to {host}: this lab stays on your machine")
        return lookup(host, *args, **kwargs)

    def guarded_connect(sock, address):
        if isinstance(address, tuple) and address[0] not in ("127.0.0.1", "::1"):
            raise ConnectionRefusedError(f"blocked a connection to {address[0]}: this lab stays on your machine")
        return connect(sock, address)

    socket.getaddrinfo, socket.socket.connect = guarded_lookup, guarded_connect


def start(model=None):
    """Start the server in the background and return its address."""
    global MODEL
    _local_only()
    MODEL = model
    server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return f"http://127.0.0.1:{server.server_address[1]}"


def tree(*fields):
    """Print the observations received so far, each under its parent, with any fields asked for."""
    ids = {span["id"] for span in SPANS}

    def walk(span, depth):
        attributes = span["attributes"]
        line = "  " * depth + f"{span['name']} ({attributes.get('langfuse.observation.type', 'span')})"
        for field in fields:
            line += f"  {field}={attributes.get('langfuse.observation.' + field)!r}"
        print(line)
        for child in SPANS:
            if child["parent"] == span["id"]:
                walk(child, depth + 1)

    for root in [span for span in SPANS if span["parent"] not in ids]:
        walk(root, 0)
Example
python compare.py

The polite prompt kept every expected phrase, because the thanks is added in front. The secretive one refused to share the A17 and B22 details, so only the refund case passed. REQUESTS shows a dataset run item created for each case, linking its trace to the dataset. The run averages were stored as scores on each dataset run, which is what Langfuse's comparison view lines up.

One experiment run over a dataset
dataset desk-ticketsWhere is my order A17?expects: shipped on 3 MarchIs B22 on its way?expects: could not find B22I want a refund for A17expects: need approvalyour taskanswer(item.input, system=...)your evaluatorcontains-expected, 1.0 or 0.0your run evaluatorthe average over the runscores in Langfuseper item, and per runruns to compareproduction 1.0, candidate 0.33
Hover or tap a piece to see what it is and which lesson built it.
Follow one run

Pick one to watch it run, step by step.

The dataset, the task, the evaluator and the run evaluator are the four pieces you supply; the run, the traces and the scores are what Langfuse keeps.

Try it yourself
  • Add a third prompt that asks for both instructions and compare.
  • Fetch the dataset with version= set to a past time and run on it.
  • Print result.format() for one run.

You understood something today that you didn't yesterday.