LangfuseLangfuse Python SDK 4.15.4 · Python 3.10+
Dashboard
0%

The gate: released, or blocked

The desk fetches whichever prompt carries the production label. Moving that label is the riskiest thing anyone does to it, so this is the step that makes it safe.

A candidate is saved as a version no desk fetches, scored beside the one in production, and takes the label only if it does as well.

Examplerelease.py, continued
def release(text):
    langfuse = get_client()
    candidate = langfuse.create_prompt(name="desk-system", prompt=text, labels=["candidate"])
    production = langfuse.get_prompt("desk-system")
    result, candidate_score = score_prompt(candidate, f"version-{candidate.version}")
    _, production_score = score_prompt(production, f"version-{production.version}")
    if candidate_score < production_score:
        raise RegressionError(result=result, metric="contains-expected", value=candidate_score,
                              threshold=production_score)
    langfuse.update_prompt(name="desk-system", version=candidate.version, new_labels=["production"])
    return candidate.version
Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports these files. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep them in the same folder.
View the code here
desk.py
"""The shop's support desk, traced: the pieces from lessons 4 to 8 in one module."""
import re

from langfuse import get_client, observe

from shop_model import reply

ORDERS = {"A17": "shipped on 3 March"}
SYSTEM = "You answer customers of a small online shop in one short sentence."


@observe(name="lookup-order", as_type="tool")
def lookup_order(order_id):
    status = ORDERS.get(order_id, "not found")
    if status == "not found":
        get_client().update_current_span(level="WARNING", status_message=f"no order {order_id}")
    return status


@observe(name="write-reply", as_type="generation")
def ask_model(messages):
    text, usage = reply(messages)
    get_client().update_current_generation(model="shop-model", usage_details=usage)
    return text


@observe(name="answer-ticket", as_type="agent")
def answer(ticket, system=SYSTEM):
    order = re.search(r"[A-Z]\d{2}", ticket)
    facts = f"\nLookup: {order.group()} {lookup_order(order.group())}" if order else ""
    return ask_model([{"role": "system", "content": system}, {"role": "user", "content": ticket + facts}])
release.py
"""Release a new desk prompt only if it does as well as production on the dataset."""
from langfuse import Evaluation, RegressionError, get_client

from desk import answer


def contains_expected(*, output, expected_output, **kwargs):
    return Evaluation(name="contains-expected", value=float(expected_output.lower() in output.lower()))


def score_prompt(prompt, run_name):
    dataset = get_client().get_dataset("desk-tickets")
    result = dataset.run_experiment(name="desk-release", run_name=run_name, evaluators=[contains_expected],
                                    task=lambda *, item, **kwargs: answer(item.input, system=prompt.compile()))
    return result, sum(e.value for row in result.item_results for e in row.evaluations) / len(dataset.items)


def release(text):
    langfuse = get_client()
    candidate = langfuse.create_prompt(name="desk-system", prompt=text, labels=["candidate"])
    production = langfuse.get_prompt("desk-system")
    result, candidate_score = score_prompt(candidate, f"version-{candidate.version}")
    _, production_score = score_prompt(production, f"version-{production.version}")
    if candidate_score < production_score:
        raise RegressionError(result=result, metric="contains-expected", value=candidate_score,
                              threshold=production_score)
    langfuse.update_prompt(name="desk-system", version=candidate.version, new_labels=["production"])
    return candidate.version
shop_model.py
"""A stand-in chat model for the shop's support desk: no key, no network."""
import re


def reply(messages):
    """Answer the last message, and count words as tokens the way a chat model reports usage."""
    rules = messages[0]["content"].lower() if messages[0]["role"] == "system" else ""
    last = messages[-1]["content"]
    lookup = re.search(r"Lookup: (\S+) (.+)", last)
    if "refund" in last.lower():
        text = "Refunds need approval, so a person will review your request."
    elif lookup and "never share" in rules:
        text = "I cannot share order details."
    elif lookup and lookup.group(2) == "not found":
        text = f"I could not find order {lookup.group(1)}. Please check the number."
    elif lookup:
        text = f"Order {lookup.group(1)} {lookup.group(2)}."
    else:
        text = "Could you send your order number?"
    if "thank" in rules:
        text = "Thanks for waiting. " + text
    used = sum(len(message["content"].split()) for message in messages)
    return text, {"input": used, "output": len(text.split())}
run_release.py
from langfuse import Langfuse

import local_langfuse

url = local_langfuse.start()
langfuse = Langfuse(public_key="pk-lf-local", secret_key="sk-lf-local", base_url=url)

from langfuse import RegressionError

from release import release

CASES = [
    ("Where is my order A17?", "shipped on 3 March"),
    ("Is B22 on its way?", "could not find order B22"),
    ("I want a refund for A17", "need approval"),
]

langfuse.create_dataset(name="desk-tickets", description="Tickets the desk must answer correctly")
for ticket, expected in CASES:
    langfuse.create_dataset_item(dataset_name="desk-tickets", input=ticket, expected_output=expected)

langfuse.create_prompt(name="desk-system", prompt="Answer in one short sentence.", labels=["production"])

for text in ["Answer in one short sentence. Thank the customer.",
             "Answer in one short sentence. Never share order details."]:
    try:
        print("released version", release(text))
    except RegressionError as error:
        print("blocked:", error)

print("production is version", langfuse.get_prompt("desk-system").version)
local_langfuse.py
"""A stand-in for the Langfuse server, on your own machine.

The Langfuse SDK sends observations to /api/public/otel/v1/traces and uses a
REST API for scores, prompts and datasets. This server answers those requests
on 127.0.0.1 and keeps what it receives in lists you can print. Given a model
function, it also answers like OpenAI's chat API. start() blocks every
connection to another computer for the rest of the program.
"""
import json
import socket
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, unquote, urlparse

from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest

SPANS, SCORES, REQUESTS = [], [], []  # observations, scores, "METHOD /path" of each request
PROMPTS, DATASETS = {}, {}             # prompt name -> its versions, dataset name -> dataset
MODEL = None                           # answers /v1/chat/completions when set
NOW = "2026-03-03T09:00:00Z"


def _value(v):
    kind = v.WhichOneof("value")
    if kind == "array_value":
        return [_value(item) for item in v.array_value.values]
    return getattr(v, kind) if kind else None


def _spans(body):
    request = ExportTraceServiceRequest()
    request.ParseFromString(body)
    for resource in request.resource_spans:
        for scope in resource.scope_spans:
            for span in scope.spans:
                SPANS.append({
                    "name": span.name, "id": span.span_id.hex(), "trace": span.trace_id.hex(),
                    "parent": span.parent_span_id.hex(), "scope": scope.scope.name,
                    "attributes": {a.key: _value(a.value) for a in span.attributes},
                })
    return {}


def _scores(batch):
    for event in batch:
        if event["type"] == "score-create":
            SCORES.append(event["body"])
    return {"successes": [{"id": e["id"], "status": 201} for e in batch], "errors": []}


def _prompt(name, query, body):
    versions = PROMPTS.setdefault(name, [])
    if body is not None:  # a new version: it takes "latest" and any labels it asks for
        labels = body.get("labels", []) + ["latest"]
        for old in versions:
            old["labels"] = [label for label in old["labels"] if label not in labels]
        versions.append({"name": name, "type": body.get("type", "text"), "prompt": body["prompt"],
                         "config": body.get("config") or {}, "labels": labels, "tags": [],
                         "version": len(versions) + 1})
        return versions[-1]
    if "version" in query:
        return versions[int(query["version"]) - 1]
    label = query.get("label", "production")
    return next(v for v in versions if label in v["labels"])


def _relabel(path, body):
    name, _, version = path.split("/")[-3:]
    versions = PROMPTS[unquote(name)]
    for old in versions:
        old["labels"] = [label for label in old["labels"] if label not in body["newLabels"]]
    versions[int(version) - 1]["labels"] += body["newLabels"]
    return versions[int(version) - 1]


def _dataset(path, query, body):
    if path == "/api/public/v2/datasets":
        DATASETS[body["name"]] = {"id": body["name"], "name": body["name"], "items": [],
                                  "metadata": body.get("metadata"), "projectId": "local",
                                  "createdAt": NOW, "updatedAt": NOW}
        return DATASETS[body["name"]]
    if path.startswith("/api/public/v2/datasets/"):
        return DATASETS[unquote(path.rsplit("/", 1)[1])]
    if path == "/api/public/dataset-items" and body is not None:
        dataset = DATASETS[body["datasetName"]]
        item = {"id": body.get("id") or f"item-{len(dataset['items']) + 1}", "status": "ACTIVE",
                "input": body.get("input"), "expectedOutput": body.get("expectedOutput"),
                "metadata": body.get("metadata"), "sourceTraceId": body.get("sourceTraceId"),
                "datasetId": dataset["id"], "datasetName": dataset["name"],
                "createdAt": NOW, "updatedAt": NOW, "mediaReferences": []}
        dataset["items"].append(item)
        return item
    if path == "/api/public/dataset-items":
        items = DATASETS[query["datasetName"]]["items"]
        return {"data": items, "meta": {"page": 1, "limit": 50, "totalItems": len(items), "totalPages": 1}}
    return {"id": "run-item", "datasetRunId": body["runName"], "datasetRunName": body["runName"],
            "datasetItemId": body["datasetItemId"], "traceId": body["traceId"],
            "observationId": body.get("observationId"), "createdAt": NOW, "updatedAt": NOW}


def _chat(body):
    text, usage = MODEL(body["messages"])
    return {"id": "chatcmpl-1", "object": "chat.completion", "created": 0, "model": body["model"],
            "choices": [{"index": 0, "finish_reason": "stop", "message": {"role": "assistant", "content": text}}],
            "usage": {"prompt_tokens": usage["input"], "completion_tokens": usage["output"],
                      "total_tokens": usage["input"] + usage["output"]}}


class Handler(BaseHTTPRequestHandler):
    def _handle(self):
        url = urlparse(self.path)
        query = {key: values[0] for key, values in parse_qs(url.query).items()}
        raw = self.rfile.read(int(self.headers.get("Content-Length") or 0))
        body = json.loads(raw) if raw and "json" in self.headers.get("Content-Type", "") else None
        REQUESTS.append(f"{self.command} {url.path}")
        try:
            if url.path.endswith("/otel/v1/traces"):
                return self._reply(200, _spans(raw))
            if url.path == "/api/public/ingestion":
                return self._reply(207, _scores(body["batch"]))
            if url.path == "/api/public/projects":
                return self._reply(200, {"data": [{"id": "local", "name": "shop-desk", "metadata": {},
                                                   "organization": {"id": "local", "name": "local"}}]})
            if url.path.startswith("/api/public/v2/prompts/") and self.command == "PATCH":
                return self._reply(200, _relabel(url.path, body))
            if url.path.startswith("/api/public/v2/prompts"):
                name = body["name"] if body else unquote(url.path.rsplit("/", 1)[1])
                return self._reply(200, _prompt(name, query, body))
            if "dataset" in url.path:
                return self._reply(200, _dataset(url.path, query, body))
            if url.path.endswith("/chat/completions") and MODEL:
                return self._reply(200, _chat(body))
        except (KeyError, StopIteration):
            pass
        self._reply(404, {"message": f"not found: {url.path}"})

    do_GET = do_POST = do_PATCH = _handle

    def _reply(self, code, data):
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps(data).encode())

    def log_message(self, *args):
        pass


def _local_only():
    """Refuse to look up or connect to any other computer, so nothing leaves by mistake."""
    lookup, connect = socket.getaddrinfo, socket.socket.connect

    def guarded_lookup(host, *args, **kwargs):
        if host not in ("127.0.0.1", "localhost", "::1"):
            raise ConnectionRefusedError(f"blocked a connection to {host}: this lab stays on your machine")
        return lookup(host, *args, **kwargs)

    def guarded_connect(sock, address):
        if isinstance(address, tuple) and address[0] not in ("127.0.0.1", "::1"):
            raise ConnectionRefusedError(f"blocked a connection to {address[0]}: this lab stays on your machine")
        return connect(sock, address)

    socket.getaddrinfo, socket.socket.connect = guarded_lookup, guarded_connect


def start(model=None):
    """Start the server in the background and return its address."""
    global MODEL
    _local_only()
    MODEL = model
    server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return f"http://127.0.0.1:{server.server_address[1]}"


def tree(*fields):
    """Print the observations received so far, each under its parent, with any fields asked for."""
    ids = {span["id"] for span in SPANS}

    def walk(span, depth):
        attributes = span["attributes"]
        line = "  " * depth + f"{span['name']} ({attributes.get('langfuse.observation.type', 'span')})"
        for field in fields:
            line += f"  {field}={attributes.get('langfuse.observation.' + field)!r}"
        print(line)
        for child in SPANS:
            if child["parent"] == span["id"]:
                walk(child, depth + 1)

    for root in [span for span in SPANS if span["parent"] not in ids]:
        walk(root, 0)

The candidate is saved with the candidate label, so it exists in Langfuse but no application fetches it. Both versions run on the same dataset. The candidate takes the production label only if its score is at least production's; otherwise RegressionError carries the numbers and the run that produced them.

Examplerun_release.py, after the setup lines
from langfuse import RegressionError

from release import release

The dataset lines from the experiments part follow, unchanged, and then two candidates: one polite, one that refuses to share order details.

Examplerun_release.py, continued
langfuse.create_prompt(name="desk-system", prompt="Answer in one short sentence.", labels=["production"])

for text in ["Answer in one short sentence. Thank the customer.",
             "Answer in one short sentence. Never share order details."]:
    try:
        print("released version", release(text))
    except RegressionError as error:
        print("blocked:", error)

print("production is version", langfuse.get_prompt("desk-system").version)
Example
python run_release.py

The polite candidate matched production and became version 2. The secretive one scored 0.33 against 1.0 and was blocked, so production stayed where it was. Both runs stay in Langfuse with their traces and scores, so someone can open the failing cases and see what the candidate said instead.

One prompt release, from candidate to label
two experiment runs on the same datasetpassesfailscandidate promptcreate_prompt, label candidateproduction promptthe version in use todaydesk-ticketsthree tickets with expectationsrun: candidatethe desk on every caserun: productionthe same cases againaverage per runcontains-expectedthe gatecandidate at least production?releasedthe label movesblockednothing moves
Hover or tap a piece to see what it is and which lesson built it.
Follow a release

Pick one to watch it run, step by step.

Click through both paths in the drawing: the release that goes through, and the one that stops at the gate.

The rest of the platform

Everything above is what the Python SDK controls. These are the parts it does not, each with where it fits.

TopicWhat it is for
LLM-as-a-judge and online evaluationEvaluators configured in Langfuse that score traces as they arrive, instead of code you write.
Annotation queues and scores in the interfacePeople reviewing traces and scoring them by hand.
The query API, metrics and dashboardsReading traces, scores and costs back out, aggregated by model, user or tag.
Prompt composability and foldersOne prompt including another, and prompts grouped by name.
Multi-modal traces and MCP tracingImages, audio and files on observations, and traces that follow a call into an MCP server.
Framework integrationsLangChain, LlamaIndex, the OpenAI Agents SDK and others, which create observations for you.
The interface, administration and self-hostingPlayground, alerts, access control, retention, and running the server yourself.
Try it yourself
  • Add a third candidate that combines both instructions and run the release.
  • Change the rule so a candidate has to beat production, not match it.
  • Add a dataset item the polite prompt fails, and see which candidates are released.

Little by little, you're building something great.