Evals in CI: failing a build on a score

Scores a person reads are scores a person forgets to read. In CI a metric is an assertion: below the line, the build goes red.

python

from pretend_ragas import PretendJudge
from ragas.metrics.collections import ContextRecall, Faithfulness
from support_bot import support_bot

judge = PretendJudge()
QUESTIONS = [("Where is order A17?", "Order A17 shipped on 3 March by courier."),
             ("How long do refunds take?", "Refunds are paid within 5 working days.")]

The top of test_support_bot.py: the stand-in judge, the bot, and the questions with the answers they should produce.

python

def test_faithfulness_and_recall():
    for question, reference in QUESTIONS:
        answer, documents = support_bot(question)
        context = documents or ["nothing was retrieved"]
        faith = Faithfulness(llm=judge).score(user_input=question, response=answer, retrieved_contexts=context)
        recall = ContextRecall(llm=judge).score(user_input=question, retrieved_contexts=context, reference=reference)
        assert faith.value >= 0.8, f"{question}: faithfulness {faith.value}"
        assert recall.value >= 0.8, f"{question}: context recall {recall.value}"

An ordinary pytest test. The only RAGAS-specific part is the two scores, and the only decision is the number you compare them against.

bash

pytest test_support_bot.py -q

text

.                                                                        [100%]
1 passed in 1.30s

The threshold is the whole design. Too high and the build breaks on wording; too low and it never catches anything. Score twenty answers you already believe in, look at the lowest, and put the line just under it.

In a pipeline

yaml

- name: Run evals
  run: |
    pip install "ragas==0.4.3" "langchain-community<0.4" pytest
    pytest test_support_bot.py -q

Any CI system that runs a shell command. Run it on pull requests that touch the retriever or the prompt, which is where regressions come from.

Keep the judge fake in CI

A stand-in judge makes the suite free, instant and repeatable, which is what a build needs. Run the real judge on a schedule instead, where cost and wobble do not block anybody.

Try it yourself

Break the retriever and watch the test fail with the score in the message.
Add a third question and give it its own threshold.

PreviousA real judge: llm_factory and an API key

Slow is fine. Stopping is the only problem.