Support bot evaluation: build the suite end to end

Lesson 0 promised an evaluation suite for one support bot, runnable with no key. This is it, and every piece is something you built alone in an earlier lesson.

import asyncio

from pretend_ragas import PretendEmbeddings, PretendJudge
from ragas import Dataset, experiment
from ragas.metrics.collections import AnswerRelevancy, ContextRecall, Faithfulness

judge, embeddings = PretendJudge(), PretendEmbeddings()
faithfulness = Faithfulness(llm=judge)
recall = ContextRecall(llm=judge)
relevancy = AnswerRelevancy(llm=judge, embeddings=embeddings)

Three metrics, one per way this bot can fail: inventing things, searching badly, and answering a different question. Judged by the stand-in from lesson 4, with the embedding model from lesson 9. The questions are the ones saved in lesson 12, in datasets/support.csv.

@experiment()
async def suite(row):
    answer, documents = support_bot(row["user_input"])
    context = documents or ["nothing was retrieved"]
    scores = {
        "faithfulness": (await faithfulness.ascore(user_input=row["user_input"], response=answer, retrieved_contexts=context)).value,
        "context_recall": (await recall.ascore(user_input=row["user_input"], retrieved_contexts=context, reference=row["reference"])).value,
        "answer_relevancy": round((await relevancy.ascore(user_input=row["user_input"], response=answer)).value, 3),
    }
    return {**row, "response": answer, **scores}

One row in, one row out, with three scores added. Every metric is awaited, because an experiment function is asynchronous.

dataset = Dataset.load(name="support", backend="local/csv", root_dir=".")
rows = asyncio.run(suite.arun(dataset, name="suite"))
for row in sorted(rows, key=lambda r: r["user_input"]):
    print(row["faithfulness"], row["context_recall"], row["answer_relevancy"], "|", row["user_input"])

Two questions, three scores each, written to experiments/suite.csv as well as printed. That file is the baseline: change the retriever, run it again, and the comparison is two files.

What this course left out

Where to go next

• Point it at your own bot. Write ten questions and references before you write any metric.
• Swap in a real judge, lesson 17, and see which scores move.
• Put it in CI, lesson 18, so a bad retriever change cannot merge quietly.
• Generate more questions with the test data generator once the hand-written ten are passing.