LLMTestCase: one answer to test

Lesson 1 compared a list of answers against one right answer, in a loop. DeepEval keeps a question and its answers in one object, a test case, and every metric reads from it.

def support_bot(question):
    return "Order A17 shipped on 3 March."

This is the bot for now: a function that returns a string. Later parts give it documents to read and a tool to call, and the test cases grow with it.

from deepeval.test_case import LLMTestCase

question = "Where is order A17?"
case = LLMTestCase(
    input=question,
    actual_output=support_bot(question),
    expected_output="Order A17 shipped on 3 March.",
)

print("input:          ", case.input)
print("actual_output:  ", case.actual_output)
print("expected_output:", case.expected_output)

input is what the user asked. The docs are specific about it: the question itself, not the prompt template your app wraps around it.

actual_output is what your app answered. In a real test you call the app to get it, as this does, rather than pasting an answer in.

expected_output is the answer you would like. Not every metric needs one, and each metric's page says whether it does.

Nothing is checked yet

Creating a test case scores nothing. It is a record of one interaction, and the fields you did not set are simply empty.

print(case.retrieval_context)
print(case.tools_called)

Each empty field arrives with the lesson that needs it. retrieval_context, the documents the bot read, comes in lesson 13. tools_called, the tools it used, in lesson 19.

The field you cannot leave out

try:
    LLMTestCase(actual_output="Order A17 shipped on 3 March.")
except Exception as error:
    print(type(error).__name__)
    print(str(error).splitlines()[1:3])

A test case without an input is never created. Everything else can be added later, but a test case has to be about a question.