Dataset: questions in a file, not in the test

Every score so far came from questions written next to the metric. A suite you run every week keeps its questions somewhere else: a file you commit, review and add to.

Example

from ragas import Dataset

dataset = Dataset(name="support", backend="local/csv", root_dir=".")
for row in [{"user_input": "Where is order A17?",
             "reference": "Order A17 shipped on 3 March by courier."},
            {"user_input": "How long do refunds take?",
             "reference": "Refunds are paid within 5 working days."}]:
    dataset.append(row)

path = dataset.save()
print(len(dataset), "rows saved")
print(open("datasets/support.csv").read())

A dataset is rows and a backend. local/csv writes them next to your code, under datasets/, which is the file your team reviews in a pull request.

Notice what the rows hold: the question, and the answer you want. Not the bot's answer. That arrives when the bot runs, which is the next lesson.

Example

loaded = Dataset.load(name="support", backend="local/csv", root_dir=".")
print(len(loaded), "rows loaded")
print(loaded[0]["user_input"], "->", loaded[0]["reference"])

Loading it back gives the same rows. From here every run starts from the file, so two runs a month apart ask exactly the same questions.

Write the questions before you read the answers

Deciding what a good answer contains, before looking at what the bot says, is what stops a test set from being a record of whatever the bot happened to do.

Try it yourself

Add a question the policy does not answer and save the file again.
Open the CSV in a spreadsheet, add a row by hand, and load it in Python.

PreviousDiscreteMetric: your own criteria in plain words

You understood something today that you didn't yesterday.