The gate: released, or blocked

The desk fetches whichever prompt carries the production label. Moving that label is the riskiest thing anyone does to it, so this is the step that makes it safe.

A candidate is saved as a version no desk fetches, scored beside the one in production, and takes the label only if it does as well.

def release(text):
    langfuse = get_client()
    candidate = langfuse.create_prompt(name="desk-system", prompt=text, labels=["candidate"])
    production = langfuse.get_prompt("desk-system")
    result, candidate_score = score_prompt(candidate, f"version-{candidate.version}")
    _, production_score = score_prompt(production, f"version-{production.version}")
    if candidate_score < production_score:
        raise RegressionError(result=result, metric="contains-expected", value=candidate_score,
                              threshold=production_score)
    langfuse.update_prompt(name="desk-system", version=candidate.version, new_labels=["production"])
    return candidate.version

The candidate is saved with the candidate label, so it exists in Langfuse but no application fetches it. Both versions run on the same dataset. The candidate takes the production label only if its score is at least production's; otherwise RegressionError carries the numbers and the run that produced them.

from langfuse import RegressionError

from release import release

The dataset lines from the experiments part follow, unchanged, and then two candidates: one polite, one that refuses to share order details.

langfuse.create_prompt(name="desk-system", prompt="Answer in one short sentence.", labels=["production"])

for text in ["Answer in one short sentence. Thank the customer.",
             "Answer in one short sentence. Never share order details."]:
    try:
        print("released version", release(text))
    except RegressionError as error:
        print("blocked:", error)

print("production is version", langfuse.get_prompt("desk-system").version)

python run_release.py

The polite candidate matched production and became version 2. The secretive one scored 0.33 against 1.0 and was blocked, so production stayed where it was. Both runs stay in Langfuse with their traces and scores, so someone can open the failing cases and see what the candidate said instead.

Click through both paths in the drawing: the release that goes through, and the one that stops at the gate.

The rest of the platform

Everything above is what the Python SDK controls. These are the parts it does not, each with where it fits.