Task completion: did the agent finish the job

Faithfulness graded the answer. Once a run has steps in it, there is a different question: did the whole run actually do what the user asked?

from deepeval.metrics import TaskCompletionMetric

for golden in dataset.evals_iterator(metrics=[TaskCompletionMetric(model=PretendJudge())],
                                     display_config=DisplayConfig(show_indicator=False)):
    support_bot(golden.input)

TaskCompletionMetric reads the trace, not a test case you built. It works out what the task was and what the outcome was from the run itself, then scores how well one matches the other.

It needs tracing, and it needs nothing else: no expected answer, no documents. That makes it the first metric in this course you could point at a live agent.

The order question passed and the refund question scored 0.33, even though the bot answered it perfectly well. That score is the stand-in judge showing its limits: it compares words, and the answer never repeats how long or take. A real judge reads it as a complete answer.

This is worth knowing in general, not only here. A judged score is an opinion from a particular judge, and when a score looks wrong the judge is the first thing to check, not the bot.

A question the bot cannot answer

missing = EvaluationDataset(goldens=[Golden(input="Where is order Z99?")])
for golden in missing.evals_iterator(metrics=[TaskCompletionMetric(model=PretendJudge())],
                                     display_config=DisplayConfig(show_indicator=False)):
    support_bot(golden.input)

Nothing in the policy mentions order Z99. The retriever from lesson 13 matches on any shared word, so the question found the line about A17, and the bot answered confidently about the wrong order.

Nothing crashed. No document was invented. The run still failed, because task completion compares what was asked with what came out, and the customer asked about a different order.

What the judge was asked

@answers("task_completion.TaskAndOutcome")
def task_and_outcome(prompt, judge):
    trace = json.loads(prompt[prompt.rindex("trace:") + 6:prompt.rindex("JSON:")])
    return {"task": " ".join(map(str, trace["input"].values())), "outcome": str(trace["output"])}

@answers("task_completion.TaskCompletionVerdict")
def task_verdict(prompt, judge):
    task, outcome = section(prompt, "Task"), section(prompt, "Actual outcome")
    return {"verdict": share(task, outcome), "reason": f"The outcome covers {share(task, outcome):.0%} of the task."}

The first question hands the judge the trace as JSON and asks for the task and the outcome. The second asks how much of the task the outcome covers, as a number from 0 to 1. This judge answers it with the same word comparison as everything else.