Scores inside an experiment, and comparing runs

An experiment that only records answers tells you nothing. Score inside it and every run leaves a row of numbers behind, which is what makes the next change measurable.

import asyncio

from pretend_ragas import PretendJudge
from ragas import Dataset, experiment
from ragas.metrics.collections import ContextRecall, Faithfulness

judge = PretendJudge()
dataset = Dataset.load(name="support", backend="local/csv", root_dir=".")

The judge and the dataset from the lessons before, and the two metrics this run grades on.

@experiment()
async def scored(row):
    answer, documents = support_bot(row["user_input"])
    context = documents or ["nothing was retrieved"]
    faith = await Faithfulness(llm=judge).ascore(user_input=row["user_input"], response=answer, retrieved_contexts=context)
    recall = await ContextRecall(llm=judge).ascore(user_input=row["user_input"], retrieved_contexts=context, reference=row["reference"])
    return {**row, "response": answer, "faithfulness": faith.value, "context_recall": recall.value}

The metrics are created once, outside the function, and called on each row. Their values go into the returned dictionary like any other column.

Note await ... ascore rather than score. An experiment function is asynchronous, and the synchronous score refuses to run inside one, with a message saying so. Every metric has both.

rows = sorted(asyncio.run(scored.arun(dataset, name="scored")), key=lambda r: r["user_input"])
print(len(rows), "rows scored")
for row in rows:
    print(row["faithfulness"], row["context_recall"], "|", row["user_input"])
print("mean faithfulness:", sum(r["faithfulness"] for r in rows) / len(rows))
print("mean context recall:", sum(r["context_recall"] for r in rows) / len(rows))

The rows are sorted before printing, because an experiment runs them at the same time and they finish in whatever order they finish. Two rows, two scores each, and the averages underneath. Those averages are the numbers you write down before changing anything, because a change is only an improvement against a number you already had.

Every run is also a CSV under experiments/, named as you named it. Comparing two runs is opening two files, or loading them in pandas if there are hundreds of rows.