contains, equals and regex

Every check so far has been contains. There are about sixty assertion types and the useful thing about the first group is that none of them costs anything: they are string comparisons, decided on your machine in no time at all.

Promptfoo calls these deterministic. They always give the same verdict for the same answer, which makes them the right tool whenever the answer has a form you can describe.

yaml

assert:
  - type: contains
    value: 3 March
  - type: icontains
    value: COURIER
  - type: regex
    value: "\d+ March"
  - type: not-contains
    value: cancelled

Four checks on one answer, in a test that asks where order A17 is. contains is case sensitive and icontains is not. regex takes a pattern, so \d+ March accepts any day. not-contains is the negative form, and most types have one: put not- in front and the verdict flips.

Example

promptfoo eval

One row, four checks, one verdict. The row passes because all four passed, and that is the rule everywhere: a test is the unit, an assertion is a vote inside it.

The one that is nearly always wrong

equals demands the exact string. It is the check people reach for first and regret.

yaml

providers:
  - file://support_bot.py
prompts:
  - "{{question}}"
tests:
  - vars:
      question: Where is order A17?
    assert:
      - type: equals
        value: Order A17 shipped on 3 March.

Example

promptfoo eval

The answer was Order A17 shipped on 3 March by courier. and the check wanted it without the last two words. A person reading both would call the answer correct. equals called it a failure, and it was right to, because that is what it was asked.

Use equals when the answer is a label, a code or a number. Use contains and regex when it is a sentence. When the thing you care about cannot be written as either, that is what part 4 is for.

The ones worth remembering

Type	Passes when	Good for
contains	the text appears	a fact that must be mentioned
icontains	it appears, ignoring case	the same, when case is noise
regex	the pattern matches	dates, ids, formats
equals	the whole answer matches	labels and codes
starts-with	the answer opens with it	a required prefix
contains-any	any one of a list appears	several acceptable wordings
contains-all	every item of a list appears	a checklist of facts
cost	the run cost less than a number	a budget
latency	it answered inside a time	a speed limit

latency and cost need real numbers to work on. A Python provider reports neither unless you return latencyMs, tokenUsage or cost from call_api, so both checks are about real providers rather than the one in this course.

Try it yourself

Change regex to \d+ April and read the failure reason in the table.
Swap contains for contains-all with a list of two facts, one of them missing.
Put not- in front of regex and predict the verdict before running.

PreviousTests in a CSV file

Slow is fine. Stopping is the only problem.