Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

Deterministic checks: contains, equals, regex

A deterministic check compares the answer to a value you already know, with no model and no key: contains, equals, regex and their kin either match the text or they do not.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

Most of what a support bot has to get right can be checked by reading the text. Did the answer mention the shipping date? Does it start with the order line? Is a refund figure in there? Promptfoo has a family of assertions for exactly these questions, and none of them calls a model, so they run anywhere, for free, and give the same verdict every time.

The bot under test

The running example is a shop support bot. It is an ordinary Python file: promptfoo calls call_api with the rendered prompt, and the function looks the question up in a two-line handbook. A question it cannot place gets a fixed miss reply.

python
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):
    asked = prompt.lower()                      # the rendered prompt is the question
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": line}
    return {"output": "I could not find that in the handbook."}

One config, several checks

The config names the bot as the provider, sends each question through the {{question}} prompt, and lists the assertions under each test. The assertion types here cover the common shapes: an exact equals, a substring contains and its case-insensitive icontains, a starts-with, a regex, and contains-all / contains-any for a set of words.

yaml
providers:
  - file://support_bot.py
prompts:
  - "{{question}}"
tests:
  - vars: { question: "Where is order A17?" }
    assert:
      - type: contains
        value: "3 March"
      - type: starts-with
        value: "Order"
      - type: regex
        value: "A[0-9]+"
  - vars: { question: "Tell me the refund window." }
    assert:
      - type: icontains
        value: "REFUNDS"
      - type: equals
        value: "Refunds take five working days."
      - type: contains-all
        value: ["Refunds", "working", "days"]
  - vars: { question: "Is the store open on Sunday?" }
    assert:
      - type: contains
        value: "shipped"
      - type: contains-any
        value: ["handbook", "manual"]

Running the checks

Run the eval. The summary counts the tests that passed and failed. The third question is one the handbook does not cover, so its contains check for the word shipped fails, and a failed test is why the whole run reports a failure.

ExampleTwo tests pass, one fails
promptfoo eval -c det.yaml --no-table

Which check decided each answer

The summary gives counts, not detail. To see the verdict per assertion, write the results to JSON with -o and read them back. A tiny reader prints each answer, then a line per check.

python
# read out.json: each answer, its checks, and any metric scores
import json

for r in json.load(open("out.json"))["results"]["results"]:
    print(r["response"]["output"], "->", "PASS" if r["success"] else "FAIL")

    for c in r["gradingResult"]["componentResults"]:
        print("  ", "pass" if c["pass"] else "FAIL", c["assertion"]["type"], "-", c["reason"])

    if r["namedScores"]:
        scores = {k: round(v, 3) for k, v in r["namedScores"].items()}
        print("   metrics", scores, "| test score", round(r["score"], 3))

Point it at the same config and every assertion shows its own pass or fail, with the reason promptfoo recorded.

ExampleThe verdict of every check, answer by answer
promptfoo eval -c det.yaml -o out.json --no-table >/dev/null 2>&1   # run, then read the JSON
python3 show.py

The deterministic assertion types

TypePasses when the output
contains / icontainsHolds the value as a substring (icontains ignores case)
equalsEquals the value exactly
starts-withBegins with the value
regexMatches the regular expression
contains-allHolds every string in the list
contains-anyHolds at least one string in the list

Where deterministic checks fit

  • A fact the answer must carry, such as an order id or a date, caught by contains or regex.
  • A fixed reply you can pin exactly, where equals proves the wording did not drift.
  • A keyword set, where contains-all asks for every term and contains-any for one of them.
Watch out. contains is a plain substring test, not a word test, so contains: "cat" passes on escalate. Reach for regex with a word boundary when a loose substring would match more than you meant.
Try it yourself
  • Add an icontains check for courier on the first answer and confirm it passes whatever the case.
  • Change the equals value by one word and watch that test fail while the substring checks still pass.
  • Swap the regex for \bcat\b on an answer that holds escalate and see the word boundary refuse the match.

Little by little, you're building something great.