Custom metrics: a decorator on a function

The built-in metrics cover what everybody needs. The things only your product cares about are usually a line of Python, and RAGAS turns a function into a metric with one decorator.

from ragas.metrics import discrete_metric


@discrete_metric(name="mentions_order", allowed_values=["yes", "no"])
def mentions_order(response: str, order_id: str) -> str:
    return "yes" if order_id in response else "no"


print(mentions_order.score(response="Order A17 shipped on 3 March.", order_id="A17").value)

discrete_metric returns one of a fixed list of values: yes or no here, pass or fail, safe or unsafe. The function takes whatever arguments you name, and you pass those by keyword to score.

from ragas.metrics import numeric_metric


@numeric_metric(name="answer_length", allowed_values=(0, 1))
def answer_length(response: str) -> float:
    return min(len(response.split()) / 30, 1.0)


print(answer_length.score(response="Order A17 shipped on 3 March.").value)

numeric_metric returns a number inside a range you declare. Thirty words counts as a full answer here, so a nine word one scores 0.3.

No sample, no judge, no network. The result has the same value as any other metric, so these sit alongside faithfulness in the same run.