Answer relevancy: the judge plus an embedding model

The metrics so far needed a judge. This one needs a judge and an embedding model, and the reason why explains how it works.

Answer relevancy asks the judge to invent the question that the answer would be a good reply to, then measures how close that invented question is to the question that was actually asked. Close means the answer was on topic.

So the judge needs an answer that writes a question and says whether the answer was evasive.

EVASIVE = {"know", "sorry", "unsure", "cannot", "unable", "idea"}


@answers("AnswerRelevanceOutput")
def question_this_answers(data, model):
    answer = data.get("response", "")
    evasive = not words(answer) or bool(words(answer) & EVASIVE)
    return {"question": sentences(answer)[0] if sentences(answer) else "",
            "noncommittal": int(evasive)}

Comparing two questions needs embeddings, so the course needs a second stand-in. This one turns text into a vector by switching on a dimension for each word that matters.

class PretendEmbeddings(BaseRagasEmbedding):
    """One dimension per word, picked by a stable hash, plus one that is always
    on so that no text embeds to nothing."""

    WIDTH = 32

    def embed_text(self, text, **kwargs):
        vector = [0.0] * self.WIDTH
        vector[0] = 1.0
        for word in words(text):
            vector[zlib.crc32(word.encode()) % (self.WIDTH - 1) + 1] = 1.0
        size = sum(v * v for v in vector) ** 0.5
        return [v / size for v in vector]

One vector per text, always the same length, and the first dimension is always on so that a text with no words that matter still embeds to something. Dividing by the size makes every vector one unit long, which is what makes the comparison a plain angle between them.

    async def aembed_text(self, text, **kwargs):
        return self.embed_text(text)

from pretend_ragas import PretendEmbeddings, PretendJudge
from ragas.metrics.collections import AnswerRelevancy

relevancy = AnswerRelevancy(llm=PretendJudge(), embeddings=PretendEmbeddings())

for answer in ["Order A17 shipped on 3 March.",
               "We sell gift cards.",
               "I do not know."]:
    result = relevancy.score(user_input="Where is order A17?", response=answer)
    print(round(result.value, 4), "|", answer)

The first answer shares most of its words with the question, so the invented question lands near it. The second is about something else entirely and scores lower.

The third scores 0 for a different reason. The judge marks an evasive answer as noncommittal, and RAGAS then scores it 0 whatever the words look like, because I do not know is never a useful reply.