experiment: running the bot over the dataset

A dataset is questions. An experiment is one run of your bot over all of them, saved so you can compare it with the next run.

POLICY = [
    "Order A17 shipped on 3 March by courier.",
    "Refunds are paid within 5 working days.",
]


def retrieve(question):
    asked = set(question.lower().strip("?").split())
    return [doc for doc in POLICY if asked & set(doc.lower().strip(".").split())]

The bot from lesson 0: two policy lines and a search that keeps any line sharing a word with the question. The experiment reads the questions saved in lesson 12 from datasets/support.csv, so run it in the same folder.

import asyncio

from ragas import Dataset, experiment


def support_bot(question):
    documents = retrieve(question)
    return (documents[0] if documents else "Sorry, I could not find that."), documents


@experiment()
async def baseline(row):
    answer, documents = support_bot(row["user_input"])
    return {**row, "response": answer, "retrieved": " | ".join(documents)}

dataset = Dataset.load(name="support", backend="local/csv", root_dir=".")
results = asyncio.run(baseline.arun(dataset, name="baseline"))
print(len(results), "rows")
for row in sorted(results, key=lambda r: r["user_input"]):
    print(row["user_input"], "->", row["response"])

@experiment() wraps a function that takes one row and returns one row. Whatever you put in the returned dictionary is a column: the answer, the documents, anything else worth keeping.

The rows come back in whatever order they finished, because the experiment runs them at the same time, so the snippet sorts them before printing.

arun runs it over every row and writes the result to experiments/baseline.csv. The name is yours, and it is what you will compare against later, so it is worth naming after the change you made.