Experiments: run the desk on every case

run_experiment runs your task on every case and passes each output to evaluators that return scores. A crashing evaluator loses its score without an error.

Lesson 25 stored the cases. An experiment runs your application on each of them and scores the results. The data can be a Langfuse dataset, as in lesson 27, or a plain list, as here.

from langfuse import Evaluation


def run_desk(*, item, **kwargs):
    return answer(item["input"])


def contains_expected(*, output, expected_output, **kwargs):
    found = expected_output.lower() in output.lower()
    return Evaluation(name="contains-expected", value=1.0 if found else 0.0)

The task is your application, called once per case with the item. An evaluator receives the input, output, expected output and metadata as keyword arguments and returns an Evaluation, which becomes a score. This one checks that the expected phrase appears in the answer.

CASES = [
    {"input": "Where is my order A17?", "expected_output": "shipped on 3 March"},
    {"input": "Is B22 on its way?", "expected_output": "could not find order B22"},
    {"input": "Hello?"},
]

result = langfuse.run_experiment(name="desk-baseline", data=CASES, task=run_desk, evaluators=[contains_expected])
for row in result.item_results:
    print(row.item["input"], "|", row.output, "|", [e.value for e in row.evaluations])

python experiment.py

Two cases passed. The third, Hello?, has no expected output, so the evaluator called .lower() on None and failed. The runner logged it and carried on, and that case has no evaluation at all. An average over the evaluations now reads 2 out of 2, a perfect score, while one case was never judged. The CI documentation calls this out: a failed evaluator must not silently leave the average.

def contains_expected(*, output, expected_output, **kwargs):
    if expected_output is None:
        return Evaluation(name="contains-expected", value=0.0, comment="no expected output to compare")
    found = expected_output.lower() in output.lower()
    return Evaluation(name="contains-expected", value=1.0 if found else 0.0)

python experiment.py

Every case now has an evaluation, and the missing expectation is visible as a 0 with a reason in its comment. A task that raises is isolated the same way: that case is logged and left out, and the others still run.

What an experiment records

python experiment.py

Each case is a trace of its own: experiment-item-run holds the task, which holds the desk's usual observations, and a separate branch holds the evaluator, recorded with the evaluator type. The scores are attached to these traces, and in Langfuse's interface the run appears under Experiments. Experiment traces carry the environment sdk-experiment, so they can be kept apart from production traffic.