DAGMetric: a decision tree judge

G-Eval hands the whole judgement to the judge and takes the score it gives. A DAG breaks the judgement into small questions, and you decide the score for every outcome.

The check: an answer about an order should name the order, and then say what happened to it. No order id is worth nothing. Shipped is full marks, delayed is 6 out of 10, lost is 2.

python

from deepeval.metrics import DAGMetric
from deepeval.metrics.dag import BinaryJudgementNode, DeepAcyclicGraph, NonBinaryJudgementNode, TaskNode
from deepeval.test_case import LLMTestCase, SingleTurnParams
from pretend_judge import PretendJudge

python

find_id = TaskNode(
    instructions="Extract the order id from the answer.",
    output_label="Order id",
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT],
)
has_id = BinaryJudgementNode(criteria="Did the previous step find an order id?")
status = NonBinaryJudgementNode(
    criteria="Does the answer say the order shipped, is delayed or is lost?",
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT],
)

Three kinds of node. A TaskNode does some work and passes its result on, here pulling the order id out of the answer. It never scores anything. A BinaryJudgementNode asks a yes or no question. A NonBinaryJudgementNode picks one of several named outcomes.

python

find_id.add_node(has_id)
has_id.add_verdict(verdict=False, score=0)
has_id.add_verdict(verdict=True, then=status)
status.add_verdict(verdict="shipped", score=10)
status.add_verdict(verdict="delayed", score=6)
status.add_verdict(verdict="lost", score=2)

dag = DeepAcyclicGraph(root_nodes=[find_id])
order_status = DAGMetric(name="Order status", dag=dag, model=PretendJudge())

add_node sends the task's result to the yes or no question. add_verdict says what each outcome leads to: either a score from 0 to 10, which ends the path, or then, another node to go to. Each verdict takes exactly one of the two.

Example

answers = ["Order A17 shipped on 3 March.", "Order A17 is delayed.",
           "Sorry, order B22 is lost.", "Your parcel is lost."]
for answer in answers:
    order_status.measure(LLMTestCase(input="Where is my order?", actual_output=answer))
    print(order_status.score, answer)

Every score is one you wrote, divided by 10. The fourth answer names no order, so the tree stopped at the first question and scored 0, even though the parcel is lost. The judge still makes each decision, but it never picks a number.

What the judge answered

Each node sends its own question. The task node asks for an output, and this judge's rule is the first word with both letters and digits in it.

python

@answers("dag.TaskNodeOutput")
def dag_task(prompt, judge):
    ids = re.findall(r"\b(?=\w*\d)(?=\w*[A-Za-z])\w+\b", section(prompt, "Actual Output"))
    return {"output": ids[0] if ids else "none"}

The yes or no question reads what the task found, which DeepEval places under the output label.

python

@answers("dag.BinaryJudgementVerdict")
def dag_yes_no(prompt, judge):
    found = prompt.split("\n**\nIMPORTANT")[0].rsplit(":\n", 1)[-1].strip()
    return {"verdict": found != "none", "reason": f"The previous step found: {found}"}

The last question lists the allowed outcomes in the prompt, and the rule picks the one the answer mentions.

python

@answers("dag.NonBinaryJudgementVerdict")
def dag_choice(prompt, judge):
    options = sorted(ast.literal_eval(re.findall(r"'verdict' (\[.*?\])", prompt)[-1]))
    text = section(prompt, "Actual Output").lower()
    chosen = next((o for o in options if o.lower() in text), options[0])
    return {"verdict": chosen, "reason": f"The answer says {chosen}."}

A tree that cannot finish

Example

check = BinaryJudgementNode(criteria="Did the previous step find an order id?")
check.add_verdict(verdict=True, score=10)
try:
    DeepAcyclicGraph(root_nodes=[check])
except Exception as error:
    print(type(error).__name__)
    print(error)

A yes or no question with only a yes branch. DeepAcyclicGraph checks the whole tree when you create it, and refuses one where a path has nowhere to go. You find out when you build the metric, not halfway through a run.

Try it yourself

Add a verdict for "refunded" with a score of 8 and an answer that says so.
Change the delayed score to 4 and predict the second answer's new score before you run it.

PreviousThreshold and strict_mode: what counts as a pass

Little by little, you're building something great.