LLM evaluation: checking an answer by hand

Before any library, test a bot's answer the way you would test a function: compare it with the right answer. It works for exactly one answer and breaks on the next.

The bot in this course answers questions about orders. Here is the answer it should give, and three answers a model could really produce.

expected = "Order A17 shipped on 3 March."
answers = [
    "Order A17 shipped on 3 March.",
    "Your order A17 was shipped on 3 March.",
    "Order A17 has not shipped. It was due on 3 March.",
]

for answer in answers:
    print(answer == expected, answer)

Only the first passes. The second says the same thing in different words and fails anyway. Models reword their answers all the time, so a check like this fails good answers again and again, and people learn to ignore it.

Looser, and wrong the other way

The obvious fix is to look only for the part that matters, the date.

for answer in answers:
    print("3 March" in answer, answer)

Now all three pass, including the third, which says the order has not shipped. The right words are there and the meaning is the opposite.

What a real check needs

• Something that reads meaning, not characters. That is a judge, from lesson 8.
• A score, not only yes or no, so an answer that is half right looks half right.
• A pass mark you choose, so you decide how good is good enough.
• The same shape for every check, so a hundred questions run like one.

DeepEval provides those four things. The next lesson is the first of them: one object that holds a question and its answers.