A Python assertion: a check that is a function

Sooner or later the rule you want is not a string comparison and not a schema. It is a paragraph of policy, and the shortest way to say it is code.

A python assertion points at a file. Promptfoo calls get_assert in it with the answer and the test's context, and reads back a verdict. It is the same idea as the provider in lesson 5, on the other side of the run.

BANNED = ["cancelled", "sorry for the inconvenience"]

def get_assert(output, context):
    found = [word for word in BANNED if word in output.lower()]
    return {
        "pass": not found,
        "score": 0.0 if found else 1.0,
        "reason": "said " + found[0] if found else "nothing banned",
    }

Two banned phrases and a check that neither appears. The return is a dictionary: whether it passed, a score between zero and one, and a reason that shows up in the table and the viewer.

tests:
  - vars:
      question: Where is order A17?
    assert:
      - type: python
        value: file://house_style.py
  - vars:
      question: Where is order B99?
    assert:
      - type: python
        value: file://house_style.py

promptfoo eval

Both rows pass, because neither answer used a banned phrase. The reason column carries the sentence the function returned, which is the part that makes a custom check usable by somebody who did not write it.

Making it fail

A check nobody has seen fail is a check nobody should trust. Add a word the bot really does say.

promptfoo eval

The first row fails and says which word it found. The second still passes, because the not-found sentence does not use it.

What a check can return

Returning True or False works when there is nothing to explain. Returning a number between zero and one makes it a score rather than a verdict, which matters in the next lesson. Returning a dictionary gives you all three, and a reason, which is nearly always worth the extra line.

There is a javascript assertion with the same shape, and lesson 9 used it inline. Use whichever language the rest of your tests are written in.