A regression gate in your tests

An experiment inside a test turns quality into a check that can fail a build. RegressionError is Langfuse's exception for a score below its threshold.

Lesson 27 printed two averages and left the decision to you. In a team, the decision should be automatic: a change that makes the desk worse fails its checks before anyone merges it.

Exampletest_gate.py

import pytest
from langfuse import Evaluation, Langfuse, RegressionError

import local_langfuse
from desk import SYSTEM, answer

CASES = [
    {"input": "Where is my order A17?", "expected_output": "shipped on 3 March"},
    {"input": "Is B22 on its way?", "expected_output": "could not find order B22"},
    {"input": "I want a refund for A17", "expected_output": "need approval"},
]

The cases are the ones you scored before, with the refund case added and an expectation on each. SYSTEM is the desk's current instruction.

Exampletest_gate.py, continued

@pytest.fixture(scope="module")
def langfuse():
    url = local_langfuse.start()
    return Langfuse(public_key="pk-lf-local", secret_key="sk-lf-local", base_url=url)

The fixture starts the local server once for the whole file.

Exampletest_gate.py, continued

def contains_expected(*, output, expected_output, **kwargs):
    return Evaluation(name="contains-expected", value=float(expected_output.lower() in output.lower()))


def gate(langfuse, system, threshold=1.0):
    result = langfuse.run_experiment(name="desk-gate", data=CASES, evaluators=[contains_expected],
                                     task=lambda *, item, **kwargs: answer(item["input"], system=system))
    values = [e.value for row in result.item_results for e in row.evaluations]
    score = sum(values) / len(CASES)
    if score < threshold:
        raise RegressionError(result=result, metric="contains-expected", value=score, threshold=threshold)
    return score

The average divides by the number of cases, not the number of evaluations, so an evaluator that crashes counts as a failure instead of vanishing from the average. Below the threshold, gate raises RegressionError with the metric, the value and the threshold.

Exampletest_gate.py, continued

def test_current_prompt_passes(langfuse):
    assert gate(langfuse, SYSTEM) == 1.0


def test_secretive_prompt_is_blocked(langfuse):
    with pytest.raises(RegressionError, match="contains-expected"):
        gate(langfuse, SYSTEM + " Never share order details.")

Example

pytest -q -p no:warnings test_gate.py

Both tests pass: the current prompt scores 1.0, and the secretive prompt is refused with a RegressionError. The tests run with no key and no network, so they fit any CI system.

Langfuse's GitHub Action

Langfuse also publishes a GitHub Action, langfuse/experiment-action, that runs an experiment script on each pull request against a dataset in Langfuse, fails the job on RegressionError, and comments the scores on the pull request. The script defines experiment(context) and calls context.run_experiment; the action needs Langfuse keys as repository secrets.

Example.github/workflows/langfuse-experiment.yml

name: Langfuse experiment gate
on: pull_request
jobs:
  experiment:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-python@v6
      - uses: langfuse/experiment-action@<release tag>
        with:
          langfuse_public_key: ${{ secrets.LANGFUSE_PUBLIC_KEY }}
          langfuse_secret_key: ${{ secrets.LANGFUSE_SECRET_KEY }}
          experiment_path: experiments/desk-gate.py
          dataset_name: desk-tickets

An average can hide a case that newly fails while two others newly pass. The documentation's stricter pattern keeps a reviewed file of which cases passed in the approved version and fails the build when any of those now fails.

Try it yourself

Lower threshold to 0.3 and run the tests again.
Add a fourth case that the current prompt fails, and read the pytest report.
Print str(error) for the RegressionError.

PreviousComparing prompts on a Langfuse dataset

Slow is fine. Stopping is the only problem.