Comparing prompts on a Langfuse dataset

dataset.run_experiment links every run to the dataset, and a run evaluator scores the run as a whole. Two runs with two prompts give two numbers to compare.

Lesson 26 ran on a list in your code. Run on a dataset stored in Langfuse, each run is recorded against the dataset, so runs made weeks apart, by different people, can be compared side by side. This lesson compares two candidate system prompts.

A model that follows instructions

A prompt comparison needs a model that reads its prompt. The stand-in from lesson 6 ignores the system message, so it learns two instructions: to thank the customer, and to never share order details.

    rules = messages[0]["content"].lower() if messages[0]["role"] == "system" else ""
    ...
    elif lookup and "never share" in rules:
        text = "I cannot share order details."
    ...
    if "thank" in rules:
        text = "Thanks for waiting. " + text

The full file is lesson 6's with these lines added: rules is the lowercased system message, the new elif sits before the not-found case, and the thank-you check runs right before the usage count. Save it; lessons from here on use this version.

Two runs on one dataset

def average(*, item_results, **kwargs):
    values = [e.value for row in item_results for e in row.evaluations]
    return Evaluation(name="average", value=sum(values) / len(values))


def compare(system, run_name):
    def task(*, item, **kwargs):
        return answer(item.input, system=system)

    result = dataset.run_experiment(name="desk-prompts", run_name=run_name, task=task,
                                    evaluators=[contains_expected], run_evaluators=[average])
    print(run_name, "average:", round(result.run_evaluations[0].value, 2))

A run evaluator receives every item's result and returns one evaluation for the whole run, here the average. The task gets a dataset item, so the ticket is item.input. run_name names each run; without one the SDK makes one from the name and the time.

dataset = langfuse.get_dataset("desk-tickets")
compare("Answer in one short sentence. Thank the customer.", "thank-the-customer")
compare("Answer in one short sentence. Never share order details.", "never-share")

langfuse.flush()
print([r for r in local_langfuse.REQUESTS if "run-items" in r])
print([(s["name"], s.get("datasetRunId"), s["value"]) for s in local_langfuse.SCORES if s.get("datasetRunId")])

python compare.py

The polite prompt kept every expected phrase, because the thanks is added in front. The secretive one refused to share the A17 and B22 details, so only the refund case passed. REQUESTS shows a dataset run item created for each case, linking its trace to the dataset. The run averages were stored as scores on each dataset run, which is what Langfuse's comparison view lines up.

The dataset, the task, the evaluator and the run evaluator are the four pieces you supply; the run, the traces and the scores are what Langfuse keeps.