LangfuseLangfuse Python SDK 4.15.4 · Python 3.10+
Dashboard
0%

A stand-in model, traced as a generation

A generation is an observation for a model call: it carries the model's name, token usage and cost. The desk gets a stand-in model you write, so no key is needed.

The desk from part 1 answered straight from its dictionary of orders. A real support desk asks a model to write the reply. Langfuse records a model call as a generation, the observation type for model calls, which carries a model name, token counts and cost. To trace one without an API key, you write the model.

A model you can trace without a key

Exampleshop_model.py
"""A stand-in chat model for the shop's support desk: no key, no network."""
import re


def reply(messages):
    """Answer the last message, and count words as tokens the way a chat model reports usage."""
    last = messages[-1]["content"]
    lookup = re.search(r"Lookup: (\S+) (.+)", last)
    if "refund" in last.lower():
        text = "Refunds need approval, so a person will review your request."
    elif lookup and lookup.group(2) == "not found":
        text = f"I could not find order {lookup.group(1)}. Please check the number."
    elif lookup:
        text = f"Order {lookup.group(1)} {lookup.group(2)}."
    else:
        text = "Could you send your order number?"
    used = sum(len(message["content"].split()) for message in messages)
    return text, {"input": used, "output": len(text.split())}

reply takes a list of chat messages, the format every chat model reads, and returns its text with a usage count, the two things a model's API sends back. It reads the last message and decides: a refund question gets the refund rule, a lookup result gets turned into a sentence, and anything else gets a request for the order number. It counts words where a real model counts tokens.

Exampletry_model.py
from shop_model import reply

print(reply([{"role": "user", "content": "Where is A17?\nLookup: A17 shipped on 3 March"}]))
print(reply([{"role": "user", "content": "I want a refund"}]))
print(reply([{"role": "user", "content": "Hello?"}]))
Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports these files. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep them in the same folder.
View the code here
local_langfuse.py
"""A stand-in for the Langfuse server, on your own machine.

The Langfuse SDK sends observations to /api/public/otel/v1/traces and uses a
REST API for scores, prompts and datasets. This server answers those requests
on 127.0.0.1 and keeps what it receives in lists you can print. Given a model
function, it also answers like OpenAI's chat API. start() blocks every
connection to another computer for the rest of the program.
"""
import json
import socket
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, unquote, urlparse

from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest

SPANS, SCORES, REQUESTS = [], [], []  # observations, scores, "METHOD /path" of each request
PROMPTS, DATASETS = {}, {}             # prompt name -> its versions, dataset name -> dataset
MODEL = None                           # answers /v1/chat/completions when set
NOW = "2026-03-03T09:00:00Z"


def _value(v):
    kind = v.WhichOneof("value")
    if kind == "array_value":
        return [_value(item) for item in v.array_value.values]
    return getattr(v, kind) if kind else None


def _spans(body):
    request = ExportTraceServiceRequest()
    request.ParseFromString(body)
    for resource in request.resource_spans:
        for scope in resource.scope_spans:
            for span in scope.spans:
                SPANS.append({
                    "name": span.name, "id": span.span_id.hex(), "trace": span.trace_id.hex(),
                    "parent": span.parent_span_id.hex(), "scope": scope.scope.name,
                    "attributes": {a.key: _value(a.value) for a in span.attributes},
                })
    return {}


def _scores(batch):
    for event in batch:
        if event["type"] == "score-create":
            SCORES.append(event["body"])
    return {"successes": [{"id": e["id"], "status": 201} for e in batch], "errors": []}


def _prompt(name, query, body):
    versions = PROMPTS.setdefault(name, [])
    if body is not None:  # a new version: it takes "latest" and any labels it asks for
        labels = body.get("labels", []) + ["latest"]
        for old in versions:
            old["labels"] = [label for label in old["labels"] if label not in labels]
        versions.append({"name": name, "type": body.get("type", "text"), "prompt": body["prompt"],
                         "config": body.get("config") or {}, "labels": labels, "tags": [],
                         "version": len(versions) + 1})
        return versions[-1]
    if "version" in query:
        return versions[int(query["version"]) - 1]
    label = query.get("label", "production")
    return next(v for v in versions if label in v["labels"])


def _relabel(path, body):
    name, _, version = path.split("/")[-3:]
    versions = PROMPTS[unquote(name)]
    for old in versions:
        old["labels"] = [label for label in old["labels"] if label not in body["newLabels"]]
    versions[int(version) - 1]["labels"] += body["newLabels"]
    return versions[int(version) - 1]


def _dataset(path, query, body):
    if path == "/api/public/v2/datasets":
        DATASETS[body["name"]] = {"id": body["name"], "name": body["name"], "items": [],
                                  "metadata": body.get("metadata"), "projectId": "local",
                                  "createdAt": NOW, "updatedAt": NOW}
        return DATASETS[body["name"]]
    if path.startswith("/api/public/v2/datasets/"):
        return DATASETS[unquote(path.rsplit("/", 1)[1])]
    if path == "/api/public/dataset-items" and body is not None:
        dataset = DATASETS[body["datasetName"]]
        item = {"id": body.get("id") or f"item-{len(dataset['items']) + 1}", "status": "ACTIVE",
                "input": body.get("input"), "expectedOutput": body.get("expectedOutput"),
                "metadata": body.get("metadata"), "sourceTraceId": body.get("sourceTraceId"),
                "datasetId": dataset["id"], "datasetName": dataset["name"],
                "createdAt": NOW, "updatedAt": NOW, "mediaReferences": []}
        dataset["items"].append(item)
        return item
    if path == "/api/public/dataset-items":
        items = DATASETS[query["datasetName"]]["items"]
        return {"data": items, "meta": {"page": 1, "limit": 50, "totalItems": len(items), "totalPages": 1}}
    return {"id": "run-item", "datasetRunId": body["runName"], "datasetRunName": body["runName"],
            "datasetItemId": body["datasetItemId"], "traceId": body["traceId"],
            "observationId": body.get("observationId"), "createdAt": NOW, "updatedAt": NOW}


def _chat(body):
    text, usage = MODEL(body["messages"])
    return {"id": "chatcmpl-1", "object": "chat.completion", "created": 0, "model": body["model"],
            "choices": [{"index": 0, "finish_reason": "stop", "message": {"role": "assistant", "content": text}}],
            "usage": {"prompt_tokens": usage["input"], "completion_tokens": usage["output"],
                      "total_tokens": usage["input"] + usage["output"]}}


class Handler(BaseHTTPRequestHandler):
    def _handle(self):
        url = urlparse(self.path)
        query = {key: values[0] for key, values in parse_qs(url.query).items()}
        raw = self.rfile.read(int(self.headers.get("Content-Length") or 0))
        body = json.loads(raw) if raw and "json" in self.headers.get("Content-Type", "") else None
        REQUESTS.append(f"{self.command} {url.path}")
        try:
            if url.path.endswith("/otel/v1/traces"):
                return self._reply(200, _spans(raw))
            if url.path == "/api/public/ingestion":
                return self._reply(207, _scores(body["batch"]))
            if url.path == "/api/public/projects":
                return self._reply(200, {"data": [{"id": "local", "name": "shop-desk", "metadata": {},
                                                   "organization": {"id": "local", "name": "local"}}]})
            if url.path.startswith("/api/public/v2/prompts/") and self.command == "PATCH":
                return self._reply(200, _relabel(url.path, body))
            if url.path.startswith("/api/public/v2/prompts"):
                name = body["name"] if body else unquote(url.path.rsplit("/", 1)[1])
                return self._reply(200, _prompt(name, query, body))
            if "dataset" in url.path:
                return self._reply(200, _dataset(url.path, query, body))
            if url.path.endswith("/chat/completions") and MODEL:
                return self._reply(200, _chat(body))
        except (KeyError, StopIteration):
            pass
        self._reply(404, {"message": f"not found: {url.path}"})

    do_GET = do_POST = do_PATCH = _handle

    def _reply(self, code, data):
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.end_headers()
        self.wfile.write(json.dumps(data).encode())

    def log_message(self, *args):
        pass


def _local_only():
    """Refuse to look up or connect to any other computer, so nothing leaves by mistake."""
    lookup, connect = socket.getaddrinfo, socket.socket.connect

    def guarded_lookup(host, *args, **kwargs):
        if host not in ("127.0.0.1", "localhost", "::1"):
            raise ConnectionRefusedError(f"blocked a connection to {host}: this lab stays on your machine")
        return lookup(host, *args, **kwargs)

    def guarded_connect(sock, address):
        if isinstance(address, tuple) and address[0] not in ("127.0.0.1", "::1"):
            raise ConnectionRefusedError(f"blocked a connection to {address[0]}: this lab stays on your machine")
        return connect(sock, address)

    socket.getaddrinfo, socket.socket.connect = guarded_lookup, guarded_connect


def start(model=None):
    """Start the server in the background and return its address."""
    global MODEL
    _local_only()
    MODEL = model
    server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    return f"http://127.0.0.1:{server.server_address[1]}"


def tree(*fields):
    """Print the observations received so far, each under its parent, with any fields asked for."""
    ids = {span["id"] for span in SPANS}

    def walk(span, depth):
        attributes = span["attributes"]
        line = "  " * depth + f"{span['name']} ({attributes.get('langfuse.observation.type', 'span')})"
        for field in fields:
            line += f"  {field}={attributes.get('langfuse.observation.' + field)!r}"
        print(line)
        for child in SPANS:
            if child["parent"] == span["id"]:
                walk(child, depth + 1)

    for root in [span for span in SPANS if span["parent"] not in ids]:
        walk(root, 0)
generation2.py
from langfuse import Langfuse, observe

import local_langfuse

url = local_langfuse.start()
langfuse = Langfuse(public_key="pk-lf-local", secret_key="sk-lf-local", base_url=url)

import re

from shop_model import reply

ORDERS = {"A17": "shipped on 3 March"}
SYSTEM = "You answer customers of a small online shop in one short sentence."


@observe(name="lookup-order", as_type="tool")
def lookup_order(order_id):
    return ORDERS.get(order_id, "not found")


@observe(name="write-reply", as_type="generation")
def ask_model(messages):
    text, usage = reply(messages)
    langfuse.update_current_generation(model="shop-model", usage_details=usage)
    return text


@observe(name="answer-ticket", as_type="agent")
def answer(ticket):
    order = re.search(r"[A-Z]\d{2}", ticket)
    facts = f"\nLookup: {order.group()} {lookup_order(order.group())}" if order else ""
    return ask_model([{"role": "system", "content": SYSTEM}, {"role": "user", "content": ticket + facts}])

print(answer("Is B22 on its way?"))

langfuse.flush()
local_langfuse.tree("usage_details")
Example
python try_model.py

Save it as shop_model.py. Every lesson from here imports it, and lesson 7 serves it behind an OpenAI-style API.

A generation around the call

Examplegeneration.py, after the setup lines with observe
import re

from shop_model import reply

ORDERS = {"A17": "shipped on 3 March"}
SYSTEM = "You answer customers of a small online shop in one short sentence."


@observe(name="lookup-order", as_type="tool")
def lookup_order(order_id):
    return ORDERS.get(order_id, "not found")

The tool is lesson 4's lookup. SYSTEM is the instruction every call to the model starts with.

Examplegeneration.py, continued
def ask_model(messages):
    with langfuse.start_as_current_observation(
        as_type="generation", name="write-reply", model="shop-model", input=messages
    ) as generation:
        text, usage = reply(messages)
        generation.update(output=text, usage_details=usage,
                          cost_details={"input": usage["input"] * 1e-6, "output": usage["output"] * 2e-6})
    return text

as_type="generation" with model and input opens the observation; update adds what came back. usage_details takes counts by kind, here input and output. cost_details takes the price in US dollars; this made-up price is one dollar per million input words and two per million output words.

Examplegeneration.py, continued
@observe(name="answer-ticket", as_type="agent")
def answer(ticket):
    order = re.search(r"[A-Z]\d{2}", ticket)
    facts = f"\nLookup: {order.group()} {lookup_order(order.group())}" if order else ""
    return ask_model([{"role": "system", "content": SYSTEM}, {"role": "user", "content": ticket + facts}])

The agent looks up the order, if the ticket has one, and passes the result to the model as a Lookup: line.

Examplegeneration.py, continued
print(answer("Where is my order A17?"))

langfuse.flush()
local_langfuse.tree()
generation = next(span for span in local_langfuse.SPANS if span["name"] == "write-reply")
for key in ["model.name", "usage_details", "cost_details"]:
    print(key, "=", generation["attributes"]["langfuse.observation." + key])
Example
python generation.py

The generation sits beside the tool, both under the agent: the agent asked for a lookup, then asked the model. Usage and cost arrived as JSON. Langfuse Cloud can also work out cost itself from the model name and a price table, but a price you send takes priority, and a model it has no price for, such as shop-model, gets a cost only if you send one.

A generation is an ordinary observation with a few more fields, filled in around one call.

What a generation records
what the generation carriesmessages insystem, ticket, lookup resultwrite-replythe observation for a model callreply outthe text your code returnsmodelshop-modelusage_detailsinput and output countscost_detailsUS dollars, or inferredinput and outputmessages, and the answer
Hover or tap a piece to see what it is and which lesson built it.
Follow one model call

Pick one to watch it run, step by step.

The same step as a decorated function

Examplegeneration.py, ask_model rewritten
@observe(name="write-reply", as_type="generation")
def ask_model(messages):
    text, usage = reply(messages)
    langfuse.update_current_generation(model="shop-model", usage_details=usage)
    return text

@observe(as_type="generation") records the call; update_current_generation adds the model and usage from inside. The messages argument becomes the input and the return value the output, as in lesson 3.

Examplegeneration.py, the new last lines
print(answer("Is B22 on its way?"))

langfuse.flush()
local_langfuse.tree("usage_details")
Example
python generation2.py

B22 is not in ORDERS, so the model was given Lookup: B22 not found and said so. Every model call from here on uses this decorated form.

Try it yourself
  • Send usage_details={"input": 10, "output": 5, "cache_read": 3} and print what arrives.
  • Add model_parameters={"temperature": 0} to the generation and find it in the attributes.
  • Ask about a refund and check which branch of reply answered.

Little by little, you're building something great.