Checking answers by hand

Before promptfoo exists, here is the job it does. A small support bot, three questions, and a loop that says whether each answer was right.

The bot is a dictionary and a loop. It is not a model, and that is on purpose: for the next few lessons the point is the testing, not the answering.

HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def answer(question):
    asked = question.lower()
    for key, line in HANDBOOK.items():
        if key in asked:
            return line
    return "I could not find that in the handbook."

print(answer("Where is order A17?"))
print(answer("Where is order B99?"))

Two questions, two answers. One was in the handbook and one was not.

Checking it

Now the test. For each question there is a phrase the answer has to contain, and the loop prints whether it was there.

checks = [("Where is order A17?", "3 March"),
          ("How long does a refund take?", "five working days"),
          ("Where is order B99?", "could not find")]

for question, wanted in checks:
    got = answer(question)
    print("pass" if wanted in got else "FAIL", "|", question)

That is an eval. A list of inputs, a rule for each one, and a report. Nothing about it needs a library.

Where it stops working

It stops working the moment the job gets real. You want the same questions run against two different models, so the loop needs a second dimension. You want forty questions, so they belong in a file rather than in the code. You want to know whether an answer was polite, which no in check can tell you. And you want last week's run to compare against, which means somewhere to keep results.

Every one of those is a small amount of work, and writing them yourself is how people end up maintaining a private test framework instead of a product. Promptfoo is that framework, already written.