Scoring a prompt on the dataset

The desk answers from a prompt. Before a new version reaches customers, it should answer the tickets you already know the answers to, and give you one number.

The desk gets one more module, beside support_desk.py: the evaluator, and a function that runs a prompt over the dataset.

Examplerelease.py

"""Release a new desk prompt only if it does as well as production on the dataset."""
from langfuse import Evaluation, RegressionError, get_client

from desk import answer


def contains_expected(*, output, expected_output, **kwargs):
    return Evaluation(name="contains-expected", value=float(expected_output.lower() in output.lower()))

The evaluator is the one from the experiments part: it checks the expected text appears in the answer, and returns 1 or 0.

Examplerelease.py, continued

def score_prompt(prompt, run_name):
    dataset = get_client().get_dataset("desk-tickets")
    result = dataset.run_experiment(name="desk-release", run_name=run_name, evaluators=[contains_expected],
                                    task=lambda *, item, **kwargs: answer(item.input, system=prompt.compile()))
    return result, sum(e.value for row in result.item_results for e in row.evaluations) / len(dataset.items)

score_prompt runs the desk over every item with a given prompt and divides by the number of items, so an item that errored counts as a failure rather than disappearing from the average.

Examplescore_once.py, after the setup and dataset lines

from release import score_prompt

langfuse.create_prompt(name="desk-system", prompt="Answer in one short sentence.", labels=["production"])
production = langfuse.get_prompt("desk-system")
result, score = score_prompt(production, "version-1")
print("scored", len(result.item_results), "tickets, average", score)

Example

python score_once.py

The prompt in production answers every ticket in the set, so it scores 1.0. That number is the bar a candidate has to meet.

Try it yourself

Add a dataset item the production prompt fails, and score it again.
Print each item's evaluation instead of the average.
Score a prompt that answers in one word.

PreviousTests for what the desk sends

Every expert started right here.