Scores from your code
A score attaches a judgement to a trace, an observation or a session: a number, a yes or no, a category or a note. Scores travel by a separate API from traces.
Part 5 made it possible to compare prompt versions, but compare them on what? Latency and cost come from the trace. Quality needs a score: a named value that your code, a person or a model judge attaches after looking at the output.
ticket = "Where is my order A17?"
with langfuse.start_as_current_observation(name="support-ticket", input=ticket) as span:
text = answer(ticket)
span.update(output=text)
span.score(name="mentions-order", value=1.0 if "A17" in text else 0.0)
span.score_trace(name="resolved", value=1, data_type="BOOLEAN")
span.score_trace(name="topic", value="shipping")
span.score_trace(name="reviewer-note", value="Correct date.", data_type="TEXT")span.score scores the observation itself; span.score_trace scores the whole trace. Without data_type, Langfuse works the type out from the value: a number is NUMERIC, a string CATEGORICAL. A yes or no must be sent as 1 or 0 with data_type="BOOLEAN", and free text needs data_type="TEXT", up to 500 characters.
langfuse.flush()
for score in local_langfuse.SCORES:
target = "observation" if score.get("observationId") else "trace"
print(score["name"], "|", repr(score["value"]), "|", score.get("dataType"), "| on the", target)
print(local_langfuse.REQUESTS)python scores.pymentions-order | 1.0 | None | on the observation resolved | 1.0 | BOOLEAN | on the trace topic | 'shipping' | None | on the trace reviewer-note | 'Correct date.' | TEXT | on the trace ['POST /api/public/otel/v1/traces', 'POST /api/public/ingestion']
Four scores, one on the observation and three on the trace. The two without a data_type were sent without one, for the server to infer. REQUESTS shows that scores went to /api/public/ingestion, a REST endpoint, while observations went to the OpenTelemetry endpoint. resolved arrived as 1.0: booleans are sent as numbers, and Langfuse's newer scores API reads them back as true or false.
langfuse.create_score(trace_id=..., name=..., value=...) does the same from anywhere, given a trace id, with observation_id to target one observation. A score can even arrive before its trace does; Langfuse links them when the trace shows up. Lesson 24 uses that for feedback that comes in later.
View the code here
"""The shop's support desk, traced: the pieces from lessons 4 to 8 in one module."""
import re
from langfuse import get_client, observe
from shop_model import reply
ORDERS = {"A17": "shipped on 3 March"}
SYSTEM = "You answer customers of a small online shop in one short sentence."
@observe(name="lookup-order", as_type="tool")
def lookup_order(order_id):
status = ORDERS.get(order_id, "not found")
if status == "not found":
get_client().update_current_span(level="WARNING", status_message=f"no order {order_id}")
return status
@observe(name="write-reply", as_type="generation")
def ask_model(messages):
text, usage = reply(messages)
get_client().update_current_generation(model="shop-model", usage_details=usage)
return text
@observe(name="answer-ticket", as_type="agent")
def answer(ticket, system=SYSTEM):
order = re.search(r"[A-Z]\d{2}", ticket)
facts = f"\nLookup: {order.group()} {lookup_order(order.group())}" if order else ""
return ask_model([{"role": "system", "content": system}, {"role": "user", "content": ticket + facts}])
"""A stand-in chat model for the shop's support desk: no key, no network."""
import re
def reply(messages):
"""Answer the last message, and count words as tokens the way a chat model reports usage."""
last = messages[-1]["content"]
lookup = re.search(r"Lookup: (\S+) (.+)", last)
if "refund" in last.lower():
text = "Refunds need approval, so a person will review your request."
elif lookup and lookup.group(2) == "not found":
text = f"I could not find order {lookup.group(1)}. Please check the number."
elif lookup:
text = f"Order {lookup.group(1)} {lookup.group(2)}."
else:
text = "Could you send your order number?"
used = sum(len(message["content"].split()) for message in messages)
return text, {"input": used, "output": len(text.split())}
"""A stand-in for the Langfuse server, on your own machine.
The Langfuse SDK sends observations to /api/public/otel/v1/traces and uses a
REST API for scores, prompts and datasets. This server answers those requests
on 127.0.0.1 and keeps what it receives in lists you can print. Given a model
function, it also answers like OpenAI's chat API. start() blocks every
connection to another computer for the rest of the program.
"""
import json
import socket
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, unquote, urlparse
from opentelemetry.proto.collector.trace.v1.trace_service_pb2 import ExportTraceServiceRequest
SPANS, SCORES, REQUESTS = [], [], [] # observations, scores, "METHOD /path" of each request
PROMPTS, DATASETS = {}, {} # prompt name -> its versions, dataset name -> dataset
MODEL = None # answers /v1/chat/completions when set
NOW = "2026-03-03T09:00:00Z"
def _value(v):
kind = v.WhichOneof("value")
if kind == "array_value":
return [_value(item) for item in v.array_value.values]
return getattr(v, kind) if kind else None
def _spans(body):
request = ExportTraceServiceRequest()
request.ParseFromString(body)
for resource in request.resource_spans:
for scope in resource.scope_spans:
for span in scope.spans:
SPANS.append({
"name": span.name, "id": span.span_id.hex(), "trace": span.trace_id.hex(),
"parent": span.parent_span_id.hex(), "scope": scope.scope.name,
"attributes": {a.key: _value(a.value) for a in span.attributes},
})
return {}
def _scores(batch):
for event in batch:
if event["type"] == "score-create":
SCORES.append(event["body"])
return {"successes": [{"id": e["id"], "status": 201} for e in batch], "errors": []}
def _prompt(name, query, body):
versions = PROMPTS.setdefault(name, [])
if body is not None: # a new version: it takes "latest" and any labels it asks for
labels = body.get("labels", []) + ["latest"]
for old in versions:
old["labels"] = [label for label in old["labels"] if label not in labels]
versions.append({"name": name, "type": body.get("type", "text"), "prompt": body["prompt"],
"config": body.get("config") or {}, "labels": labels, "tags": [],
"version": len(versions) + 1})
return versions[-1]
if "version" in query:
return versions[int(query["version"]) - 1]
label = query.get("label", "production")
return next(v for v in versions if label in v["labels"])
def _relabel(path, body):
name, _, version = path.split("/")[-3:]
versions = PROMPTS[unquote(name)]
for old in versions:
old["labels"] = [label for label in old["labels"] if label not in body["newLabels"]]
versions[int(version) - 1]["labels"] += body["newLabels"]
return versions[int(version) - 1]
def _dataset(path, query, body):
if path == "/api/public/v2/datasets":
DATASETS[body["name"]] = {"id": body["name"], "name": body["name"], "items": [],
"metadata": body.get("metadata"), "projectId": "local",
"createdAt": NOW, "updatedAt": NOW}
return DATASETS[body["name"]]
if path.startswith("/api/public/v2/datasets/"):
return DATASETS[unquote(path.rsplit("/", 1)[1])]
if path == "/api/public/dataset-items" and body is not None:
dataset = DATASETS[body["datasetName"]]
item = {"id": body.get("id") or f"item-{len(dataset['items']) + 1}", "status": "ACTIVE",
"input": body.get("input"), "expectedOutput": body.get("expectedOutput"),
"metadata": body.get("metadata"), "sourceTraceId": body.get("sourceTraceId"),
"datasetId": dataset["id"], "datasetName": dataset["name"],
"createdAt": NOW, "updatedAt": NOW, "mediaReferences": []}
dataset["items"].append(item)
return item
if path == "/api/public/dataset-items":
items = DATASETS[query["datasetName"]]["items"]
return {"data": items, "meta": {"page": 1, "limit": 50, "totalItems": len(items), "totalPages": 1}}
return {"id": "run-item", "datasetRunId": body["runName"], "datasetRunName": body["runName"],
"datasetItemId": body["datasetItemId"], "traceId": body["traceId"],
"observationId": body.get("observationId"), "createdAt": NOW, "updatedAt": NOW}
def _chat(body):
text, usage = MODEL(body["messages"])
return {"id": "chatcmpl-1", "object": "chat.completion", "created": 0, "model": body["model"],
"choices": [{"index": 0, "finish_reason": "stop", "message": {"role": "assistant", "content": text}}],
"usage": {"prompt_tokens": usage["input"], "completion_tokens": usage["output"],
"total_tokens": usage["input"] + usage["output"]}}
class Handler(BaseHTTPRequestHandler):
def _handle(self):
url = urlparse(self.path)
query = {key: values[0] for key, values in parse_qs(url.query).items()}
raw = self.rfile.read(int(self.headers.get("Content-Length") or 0))
body = json.loads(raw) if raw and "json" in self.headers.get("Content-Type", "") else None
REQUESTS.append(f"{self.command} {url.path}")
try:
if url.path.endswith("/otel/v1/traces"):
return self._reply(200, _spans(raw))
if url.path == "/api/public/ingestion":
return self._reply(207, _scores(body["batch"]))
if url.path == "/api/public/projects":
return self._reply(200, {"data": [{"id": "local", "name": "shop-desk", "metadata": {},
"organization": {"id": "local", "name": "local"}}]})
if url.path.startswith("/api/public/v2/prompts/") and self.command == "PATCH":
return self._reply(200, _relabel(url.path, body))
if url.path.startswith("/api/public/v2/prompts"):
name = body["name"] if body else unquote(url.path.rsplit("/", 1)[1])
return self._reply(200, _prompt(name, query, body))
if "dataset" in url.path:
return self._reply(200, _dataset(url.path, query, body))
if url.path.endswith("/chat/completions") and MODEL:
return self._reply(200, _chat(body))
except (KeyError, StopIteration):
pass
self._reply(404, {"message": f"not found: {url.path}"})
do_GET = do_POST = do_PATCH = _handle
def _reply(self, code, data):
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.end_headers()
self.wfile.write(json.dumps(data).encode())
def log_message(self, *args):
pass
def _local_only():
"""Refuse to look up or connect to any other computer, so nothing leaves by mistake."""
lookup, connect = socket.getaddrinfo, socket.socket.connect
def guarded_lookup(host, *args, **kwargs):
if host not in ("127.0.0.1", "localhost", "::1"):
raise ConnectionRefusedError(f"blocked a connection to {host}: this lab stays on your machine")
return lookup(host, *args, **kwargs)
def guarded_connect(sock, address):
if isinstance(address, tuple) and address[0] not in ("127.0.0.1", "::1"):
raise ConnectionRefusedError(f"blocked a connection to {address[0]}: this lab stays on your machine")
return connect(sock, address)
socket.getaddrinfo, socket.socket.connect = guarded_lookup, guarded_connect
def start(model=None):
"""Start the server in the background and return its address."""
global MODEL
_local_only()
MODEL = model
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
threading.Thread(target=server.serve_forever, daemon=True).start()
return f"http://127.0.0.1:{server.server_address[1]}"
def tree(*fields):
"""Print the observations received so far, each under its parent, with any fields asked for."""
ids = {span["id"] for span in SPANS}
def walk(span, depth):
attributes = span["attributes"]
line = " " * depth + f"{span['name']} ({attributes.get('langfuse.observation.type', 'span')})"
for field in fields:
line += f" {field}={attributes.get('langfuse.observation.' + field)!r}"
print(line)
for child in SPANS:
if child["parent"] == span["id"]:
walk(child, depth + 1)
for root in [span for span in SPANS if span["parent"] not in ids]:
walk(root, 0)
Scores on traces that were not sampled
The sampling documentation says that when a trace is not sampled, none of its observations or scores are sent. Lesson 16's setup tests that.
from langfuse import Langfuse
import local_langfuse
url = local_langfuse.start()
langfuse = Langfuse(public_key="pk-lf-local", secret_key="sk-lf-local", base_url=url, sample_rate=0.3)
for number in range(1, 11):
trace_id = langfuse.create_trace_id(seed=f"ticket-{number}")
with langfuse.start_as_current_observation(name="support-ticket", trace_context={"trace_id": trace_id}):
langfuse.create_score(trace_id=trace_id, name="resolved", value=1, data_type="BOOLEAN")
langfuse.flush()
print(len({span["trace"] for span in local_langfuse.SPANS}), "traces sent,", len(local_langfuse.SCORES), "scores sent")python sampled_scores.py1 traces sent, 10 scores sent
One trace was sent, as in lesson 16, but all ten scores were. In 4.15.4 the SDK's check reads a field that is always empty at that point, so every score is treated as in the sample. The nine scores for missing traces are harmless noise, but if a score carries something sensitive in its comment, sampling does not keep it back. The installed version is what runs, so plan for it.
- Send
value=Truewithdata_type="BOOLEAN"and print the value that arrives. - Score the session with
langfuse.create_score(session_id="chat-0042", ...)and print which ids the score carries. - Add a
comment=to one score and print it.
Slow is fine. Stopping is the only problem.