Support bot evaluation: build the suite end to end

Lesson 0 promised a test suite for one support bot. This is it, in three pieces, each of them something you built alone in an earlier lesson.

from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric, GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams
from pretend_judge import PretendJudge

helpful = GEval(
    name="Helpful",
    evaluation_steps=["Check that the answer contains the facts of the expected answer."],
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
    model=PretendJudge(),
)
metrics = [helpful, AnswerRelevancyMetric(model=PretendJudge()), FaithfulnessMetric(model=PretendJudge())]

Three metrics, which is inside the five the docs recommend. One written in your own words with G-Eval and steps from lesson 9, and two for a bot that reads documents, from lessons 13 and 14. All three judged by the stand-in from lesson 8.

from deepeval.dataset import EvaluationDataset, Golden

dataset = EvaluationDataset(goldens=[
    Golden(input="Where is order A17?", expected_output="Order A17 shipped on 3 March by courier."),
    Golden(input="How long do refunds take?", expected_output="Refunds are paid within 5 working days."),
    Golden(input="Where is order Z99?",
           expected_output="Sorry, I could not find anything about that: Where is order Z99?"),
])
for golden in dataset.goldens:
    answer, documents = support_bot(golden.input)
    dataset.add_test_case(LLMTestCase(input=golden.input, actual_output=answer,
                                      expected_output=golden.expected_output,
                                      retrieval_context=documents))

A dataset, from lesson 16. Three goldens: a question the policy answers, another one, and a question about an order that does not exist. Then the loop that runs the bot over them and builds the test cases.

from deepeval import evaluate
from deepeval.evaluate import DisplayConfig

result = evaluate(test_cases=dataset.test_cases, metrics=metrics,
                  display_config=DisplayConfig(print_results=False, show_indicator=False))
for test in result.test_results:
    print(("PASS" if test.success else "FAIL"), test.input)
    for data in test.metrics_data:
        print("   ", data.name, data.score, data.reason)

A report. Two questions pass. The Z99 question fails, and the reasons say why: the bot answered about order A17, so the answer is nothing to do with what was asked.

That failure is real, and the suite found it without anybody reading an answer. The retriever keeps any policy line that shares a word with the question, and order is a word.

Fix the bot, run it again

IGNORE = {"where", "is", "order", "how", "long", "do", "take", "my"}


def retrieve(question):
    asked = {word for word in question.lower().strip("?").split() if word not in IGNORE}
    return [doc for doc in POLICY if asked & set(doc.lower().strip(".").split())]


def support_bot(question):
    documents = retrieve(question)
    if documents:
        return documents[0], documents
    return f"Sorry, I could not find anything about that: {question}", documents

Two changes. The search ignores the words every question contains, so order stops matching everything. And when nothing is found the bot says so, repeating the question, which is what a customer needs to read and what the golden expects.

dataset.test_cases.clear()
for golden in dataset.goldens:
    answer, documents = support_bot(golden.input)
    dataset.add_test_case(LLMTestCase(input=golden.input, actual_output=answer,
                                      expected_output=golden.expected_output,
                                      retrieval_context=documents))
result = evaluate(test_cases=dataset.test_cases, metrics=metrics,
                  display_config=DisplayConfig(print_results=False, show_indicator=False))
print([(test.input, test.success) for test in result.test_results])

All three pass. The Z99 question finds no document, the bot says so, and faithfulness is happy because an answer with no documents behind it contradicts nothing. This is the loop the whole course was for: run the suite, read the failure, change the code, run it again.

What this course left out

DeepEval is much larger than these 23 lessons. Everything below is real, documented, and deliberately skipped, so you know it exists when you need it.

Where to go next

• Point it at your own app. Write five goldens for it before you write any metric.
• Swap the judge for a real model, lesson 21, and see which scores move.
• Put it in CI, lesson 20, so nobody has to remember to run it.
• Add one safety metric from the table above, if your bot talks to customers.