LLM tracing: @observe and evals_iterator

Lesson 16 built each test case by hand: call the bot, collect what it returned, pass it in. If the bot has steps inside it, that gets awkward fast. Tracing lets the run report itself.

from deepeval.tracing import observe, update_current_trace


@observe(type="retriever")
def retrieve_traced(question):
    return retrieve(question)


@observe(type="agent")
def support_bot(question):
    documents = retrieve_traced(question)
    answer = documents[0] if documents else "Sorry, I could not find that."
    update_current_trace(input=question, output=answer, retrieval_context=documents)
    return answer

@observe on a function makes every call to it a span. The outermost call becomes the trace for that run. Nothing else about the functions changed.

update_current_trace fills in the test case from inside the run: the question, the answer, and the documents. The same fields as LLMTestCase, with actual_output called output.

print(support_bot("Where is order A17?"))

The first line is DeepEval saying it has nowhere to send the trace, because this course never logs in to Confident AI. Set CONFIDENT_TRACE_VERBOSE=0 in your environment and it goes away.

Called normally, the traced bot behaves exactly as before, and no metric runs. Tracing only evaluates when an evaluation asks it to, so this is safe to leave in code that runs in production.

Running the dataset through it

from deepeval.dataset import EvaluationDataset, Golden
from deepeval.evaluate import DisplayConfig
from deepeval.metrics import FaithfulnessMetric
from pretend_judge import PretendJudge

dataset = EvaluationDataset(goldens=[Golden(input="Where is order A17?"),
                                     Golden(input="How long do refunds take?")])
for golden in dataset.evals_iterator(metrics=[FaithfulnessMetric(model=PretendJudge())],
                                     display_config=DisplayConfig(show_indicator=False)):
    support_bot(golden.input)

evals_iterator hands you one golden at a time and collects the trace your bot produced for each. There is no add_test_case and no list of test cases: the metrics score what the trace reported.

Two goldens, two traces, two faithfulness scores, both 1. The goldens here carry no expected answer, and faithfulness does not need one.

Spans are worth naming with type: retriever, tool, llm or agent. The label does not change any score. It makes a trace readable, and it is what the next two lessons grade.