Scores from your code

A score attaches a judgement to a trace, an observation or a session: a number, a yes or no, a category or a note. Scores travel by a separate API from traces.

Part 5 made it possible to compare prompt versions, but compare them on what? Latency and cost come from the trace. Quality needs a score: a named value that your code, a person or a model judge attaches after looking at the output.

ticket = "Where is my order A17?"
with langfuse.start_as_current_observation(name="support-ticket", input=ticket) as span:
    text = answer(ticket)
    span.update(output=text)
    span.score(name="mentions-order", value=1.0 if "A17" in text else 0.0)
    span.score_trace(name="resolved", value=1, data_type="BOOLEAN")
    span.score_trace(name="topic", value="shipping")
    span.score_trace(name="reviewer-note", value="Correct date.", data_type="TEXT")

span.score scores the observation itself; span.score_trace scores the whole trace. Without data_type, Langfuse works the type out from the value: a number is NUMERIC, a string CATEGORICAL. A yes or no must be sent as 1 or 0 with data_type="BOOLEAN", and free text needs data_type="TEXT", up to 500 characters.

langfuse.flush()
for score in local_langfuse.SCORES:
    target = "observation" if score.get("observationId") else "trace"
    print(score["name"], "|", repr(score["value"]), "|", score.get("dataType"), "| on the", target)
print(local_langfuse.REQUESTS)

python scores.py

Four scores, one on the observation and three on the trace. The two without a data_type were sent without one, for the server to infer. REQUESTS shows that scores went to /api/public/ingestion, a REST endpoint, while observations went to the OpenTelemetry endpoint. resolved arrived as 1.0: booleans are sent as numbers, and Langfuse's newer scores API reads them back as true or false.

langfuse.create_score(trace_id=..., name=..., value=...) does the same from anywhere, given a trace id, with observation_id to target one observation. A score can even arrive before its trace does; Langfuse links them when the trace shows up. Lesson 24 uses that for feedback that comes in later.

Scores on traces that were not sampled

The sampling documentation says that when a trace is not sampled, none of its observations or scores are sent. Lesson 16's setup tests that.

from langfuse import Langfuse

import local_langfuse

url = local_langfuse.start()
langfuse = Langfuse(public_key="pk-lf-local", secret_key="sk-lf-local", base_url=url, sample_rate=0.3)

for number in range(1, 11):
    trace_id = langfuse.create_trace_id(seed=f"ticket-{number}")
    with langfuse.start_as_current_observation(name="support-ticket", trace_context={"trace_id": trace_id}):
        langfuse.create_score(trace_id=trace_id, name="resolved", value=1, data_type="BOOLEAN")

langfuse.flush()
print(len({span["trace"] for span in local_langfuse.SPANS}), "traces sent,", len(local_langfuse.SCORES), "scores sent")

python sampled_scores.py

One trace was sent, as in lesson 16, but all ten scores were. In 4.15.4 the SDK's check reads a field that is always empty at that point, so every score is treated as in the sample. The nine scores for missing traces are harmless noise, but if a score carries something sensitive in its comment, sampling does not keep it back. The installed version is what runs, so plan for it.