InstructorBaseRagasLLM: the stand-in judge

Every metric from here on asks a model questions. This lesson writes the model, so the rest of the course needs no key and no network.

A RAGAS judge has one method that matters: generate(prompt, response_model). The prompt is text, the response model is a Pydantic class, and the judge must return an instance of it. Anything that does that and inherits InstructorBaseRagasLLM is a model as far as RAGAS is concerned.

Reading the prompt

RAGAS prompts are long, but they all end the same way: the example section, then the real input as JSON. So the judge does not need to understand English, only to find that block.

def asked(prompt):
    """The JSON block a RAGAS prompt ends with: the input it wants judged.

    A metric you wrote yourself has no such block, so the prompt itself is the
    text to judge."""
    block = prompt.rsplit("input:", 1)[-1].rsplit("Output:", 1)[0].strip()
    try:
        return json.loads(block)
    except ValueError:
        return {"response": prompt}

Then two helpers for deciding things. The first keeps the words that carry meaning.

def words(text):
    """The words that carry meaning, lower case, without the common ones."""
    return {w for w in re.findall(r"[a-z0-9]+", str(text).lower()) if len(w) > 2 and w not in SKIP}

The second is the rule almost every answer below is built on: a claim counts as supported when every word that matters in it appears in the context.

def supported(claim, context):
    """True when every word that matters in the claim is also in the context."""
    need = words(claim)
    return bool(need) and need <= words(context)

One answer per question

A metric does not ask one big question. It asks several small ones, each with its own response model, so the judge keeps one answer per model name in a dictionary.

ANSWERS = {}


def answers(name):
    """Register one answer, under the name of the model RAGAS asks for."""
    def keep(answer):
        ANSWERS[name] = answer
        return answer
    return keep

The judge

class PretendJudge(InstructorBaseRagasLLM):
    def generate(self, prompt, response_model):
        name = response_model.__name__
        if name not in ANSWERS:
            raise NotImplementedError(f"The pretend judge has no answer for {name} yet")
        return response_model(**ANSWERS[name](asked(prompt), response_model))

    async def agenerate(self, prompt, response_model):
        return self.generate(prompt, response_model)

That is the whole class. It looks up the answer for the model it was handed, calls it with the prompt's input block, and builds the object. A question with no answer raises, naming itself, which is how you find out what a new metric wants.

Now the first answer. Faithfulness opens by asking for the statements in an answer, and splitting on full stops is close enough for a stand-in.

@answers("StatementGeneratorOutput")
def split_the_answer(data, model):
    return {"statements": sentences(data.get("answer", ""))}

from pretend_ragas import PretendJudge
from ragas.metrics.collections.faithfulness.util import StatementGeneratorOutput

judge = PretendJudge()
prompt = """Now perform the same with the following input
input: {
    "question": "Where is order A17?",
    "answer": "Order A17 shipped on 3 March. It was sent by courier."
}
Output: """
print(judge.generate(prompt, StatementGeneratorOutput).statements)

That is the first thing faithfulness asks: break this answer into statements. Two sentences in, two statements out, and no model involved.

from ragas.metrics.collections.answer_relevancy.util import AnswerRelevanceOutput
from pydantic import BaseModel


class Whatever(BaseModel):
    verdict: int


try:
    judge.generate("input: {}\nOutput: ", Whatever)
except NotImplementedError as error:
    print(error)

And a question it cannot answer. Each of the next five lessons adds one branch and explains the rule behind it.