Linking a prompt to its generations

Passing the prompt object to a generation records its name and version on the trace. Langfuse then compares latency, cost and quality per prompt version.

A version is only useful if you can tell how it did. That needs every generation to say which prompt version produced it, which the SDK does when you hand it the prompt.

langfuse.create_prompt(
    name="desk-system",
    type="text",
    prompt="You answer customers of a small online shop in one short sentence.",
    labels=["production"],
)

The desk's prompt, with the shop filled in so it has no variables.

prompt = langfuse.get_prompt("desk-system")


@observe(name="write-reply", as_type="generation")
def ask_model(messages):
    text, usage = reply(messages)
    langfuse.update_current_generation(model="shop-model", usage_details=usage, prompt=prompt)
    return text


ask_model([{"role": "system", "content": prompt.compile()}, {"role": "user", "content": "Hello?"}])
langfuse.flush()
attributes = local_langfuse.SPANS[0]["attributes"]
print({key: value for key, value in attributes.items() if ".prompt." in key})

python linked.py

The generation carries the prompt's name and version as two attributes. That link is what Langfuse's metrics for each prompt version are built from: median latency, tokens and cost, and the scores that part 6 adds. With the OpenAI wrapper from lesson 7 the same link is one keyword, langfuse_prompt=prompt, on the chat call; the project in lesson 30 uses it.

Only generations are linked, and a fallback prompt is never linked, because it is not a version stored in Langfuse. propagate_attributes(prompt=prompt) links every generation started inside it, for code where the generation is created by another library.

Two versions at once

Linking also makes an A/B test possible: two versions carry two labels, each request picks one, and Langfuse compares their metrics.

import random

langfuse.create_prompt(name="desk-ab", prompt="Answer in one short sentence.", labels=["prod-a"])
langfuse.create_prompt(name="desk-ab", prompt="Answer briefly. Thank the customer.", labels=["prod-b"])
variants = [langfuse.get_prompt("desk-ab", label="prod-a"), langfuse.get_prompt("desk-ab", label="prod-b")]

random.seed(3)
for ticket in ["Where is A17?", "Is B22 late?", "Hello?", "Where is A17 now?"]:
    prompt = random.choice(variants)
    print(ticket, "-> version", prompt.version)

python ab_test.py

Each ticket used one of the two versions, chosen at random; random.seed only makes the choice repeat here. Linked to the generations, the two groups of traces can then be compared. The documentation suggests A/B tests after a version has passed tests on a dataset, which part 6 builds.