Datasets: goldens, files and test cases

Every test case so far was written next to the metric that scored it. A real suite keeps the questions in one place, runs the bot over them, and does that again next week. That list is a dataset.

POLICY = [
    "Order A17 shipped on 3 March by courier.",
    "Refunds are paid within 5 working days.",
]


def retrieve(question):
    asked = set(question.lower().strip("?").split())
    return [doc for doc in POLICY if asked & set(doc.lower().strip(".").split())]

def support_bot(question):
    documents = retrieve(question)
    answer = documents[0] if documents else "Sorry, I could not find that."
    return answer, documents

The retriever from lesson 13, and the bot that answers from the first document it finds. It returns the documents as well, because the test case needs them.

Goldens are questions, not answers

from deepeval.dataset import EvaluationDataset, Golden

dataset = EvaluationDataset(goldens=[
    Golden(input="Where is order A17?",
           expected_output="Order A17 shipped on 3 March by courier."),
    Golden(input="How long do refunds take?",
           expected_output="Refunds are paid within 5 working days."),
])

print(len(dataset.goldens), "goldens,", len(dataset.test_cases), "test cases")
print(dataset.goldens[0].input, "|", dataset.goldens[0].actual_output)

A golden is a question, and usually the answer you would like. What it does not have is an actual_output, because your bot has not run yet. That is the whole idea: the dataset is the part that stays the same while the bot changes.

A dataset holds goldens and test cases separately. It starts with two goldens and no test cases.

Keeping it in a file

import json

path = dataset.save_as(file_type="json", directory="data", file_name="support")
print(path)
print([golden["input"] for golden in json.load(open(path))])

save_as writes the goldens out and returns the path. JSON, CSV and JSONL are the three formats. This is the file you commit, so a run today and a run in a month ask the same questions.

loaded = EvaluationDataset()
loaded.add_goldens_from_json_file(file_path="data/support.json")
print(len(loaded.goldens), loaded.goldens[1].expected_output)

Loading it back gives the same goldens. From here a run always starts with the file, not with test cases written in the test itself.

From goldens to test cases

from deepeval.test_case import LLMTestCase

for golden in loaded.goldens:
    answer, documents = support_bot(golden.input)
    loaded.add_test_case(LLMTestCase(input=golden.input, actual_output=answer,
                                     expected_output=golden.expected_output,
                                     retrieval_context=documents))
print(len(loaded.test_cases), "test cases from", len(loaded.goldens), "goldens")

This is the loop every eval run has: take each golden, call the bot on its input, and add a test case with what the bot said. The golden brings the question and the expected answer; the bot brings the answer and the documents it read.

from deepeval import evaluate
from deepeval.evaluate import DisplayConfig
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric
from pretend_judge import PretendJudge

result = evaluate(
    test_cases=loaded.test_cases,
    metrics=[AnswerRelevancyMetric(model=PretendJudge()), FaithfulnessMetric(model=PretendJudge())],
    display_config=DisplayConfig(print_results=False, show_indicator=False),
)
for test in result.test_results:
    print(test.input, [(data.name, data.score) for data in test.metrics_data])

Two questions, two metrics, one run. The refund question scores 1 on both. The order question is relevant and faithful too, because the bot answers by handing back the document it found.