DiscreteMetric: your own criteria in plain words

Lesson 10's metrics counted things. This one asks the judge a question you wrote, which is how you grade something no formula can catch: tone, safety, whether an answer sounds like your company.

from pretend_ragas import PretendJudge
from ragas.metrics import DiscreteMetric

judge = PretendJudge()
polite = DiscreteMetric(
    name="politeness",
    allowed_values=["polite", "blunt"],
    prompt="Is the response polite? Answer polite or blunt.\n\nResponse: {response}",
)
for answer in ["Thanks for waiting. Order A17 shipped on 3 March.", "Shipped. Nothing else."]:
    print(polite.score(response=answer, llm=judge).value, "|", answer)

allowed_values is the list the judge must choose from, and prompt is your question with the sample's fields in braces. RAGAS fills them in, sends it, and refuses anything outside your list. The judge is passed when you score, not when you build the metric.

What the judge was asked

COURTESY = {"thanks", "thank", "please", "sorry", "welcome", "happy"}


@answers("DiscreteResponseModel")
def pick_an_allowed_value(data, model):
    """A metric you wrote: the allowed answers live on the model RAGAS passes in."""
    import typing
    allowed = list(typing.get_args(model.model_fields["value"].annotation))
    polite = bool(words(data.get("response", "")) & COURTESY)
    return {"value": allowed[0] if polite else allowed[-1], "reason": "looked for courteous words"}

Your metric's allowed values arrive on the model itself, so the judge reads them off it rather than guessing. This rule looks for courteous words, which is exactly as clever as a stand-in should be.

This is the same idea as the aspect critics in the docs, which score harmfulness, maliciousness, coherence and conciseness. Each is a DiscreteMetric with a different sentence in it.