Tool correctness: did it call the right tool

An agent that answers from memory instead of looking something up will often sound right. The cheapest way to catch it is to check which tools it called.

from deepeval.metrics import ToolCorrectnessMetric
from pretend_judge import PretendJudge
from deepeval.test_case import LLMTestCase, ToolCall

used_the_search = LLMTestCase(
    input="Where is order A17?",
    actual_output="Order A17 shipped on 3 March by courier.",
    tools_called=[ToolCall(name="retrieve")],
    expected_tools=[ToolCall(name="retrieve")],
)
tools = ToolCorrectnessMetric(model=PretendJudge())
tools.measure(used_the_search)
print(tools.score, tools.is_successful())
print(tools.reason)

tools_called is what the run actually used, expected_tools is what it should have used, and both are lists of ToolCall. The bot searched, which is what the question needed, so the score is 1.

This metric is not judged by a model. It compares the two lists and counts, which is why the reason reads like a tally rather than an opinion. It still wants a model when you create it, and passing the stand-in judge is the simplest way to satisfy that: nothing ever asks it anything.

An agent that did not bother

guessed = LLMTestCase(
    input="Where is order A17?",
    actual_output="I think it shipped last week.",
    tools_called=[],
    expected_tools=[ToolCall(name="retrieve")],
)
tools.measure(guessed)
print(tools.score, tools.is_successful())
print(tools.reason)

No tools called at all, and a confident answer. Every metric so far might have liked that sentence. This one fails it, because the run never looked anything up.

An agent that did too much

too_many = LLMTestCase(
    input="Where is order A17?",
    actual_output="Order A17 shipped on 3 March by courier.",
    tools_called=[ToolCall(name="retrieve"), ToolCall(name="refund")],
    expected_tools=[ToolCall(name="retrieve")],
)
tools.measure(too_many)
print(tools.score, tools.is_successful())

The search was called, and so was a refund tool nobody asked for, and the score is still 1. By default the metric asks one question: was every expected tool called? An extra call is not a failure, which matters, because an agent that also refunded something would pass this check.

exact = ToolCorrectnessMetric(model=PretendJudge(), should_exact_match=True)
exact.measure(too_many)
print(exact.score, exact.is_successful())

should_exact_match requires the two lists to be the same, so now the extra call fails. should_consider_ordering is the middle setting: the same tools, in the order you expected.