What you are going to build

RAGAS scores a RAG application: whether the answer stuck to the documents, and whether the search found the right documents in the first place. This course builds an evaluation suite for one support bot, and every lesson runs on your machine with no API key.

A RAG bot answers in two steps. It searches for documents, then writes an answer from them. When the answer is wrong, either the search missed or the writing drifted, and a single score cannot tell you which. RAGAS has separate metrics for each half, which is the whole point of it.

A metric, working

from pretend_ragas import PretendJudge
from ragas.metrics.collections import Faithfulness

faithfulness = Faithfulness(llm=PretendJudge())
context = ["Order A17 shipped on 3 March by courier."]

for answer in ["Order A17 shipped on 3 March by courier.", "Order A17 was cancelled."]:
    result = faithfulness.score(user_input="Where is order A17?", response=answer,
                                retrieved_contexts=context)
    print(result.value, answer)

One metric, two answers, two scores. The first answer only says what the document says, so faithfulness is 1. The second contradicts it and scores 0.

PretendJudge is a stand-in model you write in lesson 4. Almost every RAGAS metric asks a model to judge, and by default that is an OpenAI model with a key; the stand-in decides by comparing words instead, so the whole course runs offline.

What you will have built

What you need

pip install "ragas==0.4.3" "langchain-community<0.4"

The pin matters. RAGAS 0.4.3 imports a class that langchain-community removed in its 0.4 release, so the newest of both together will not import at all. Lesson 16 says more about the version churn in this library.