Transforms: reshaping output before a check

A transform rewrites the answer before a check reads it: pull a field out of a JSON reply, or lower-case the text, so the assertion sees the shape it expects.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

Sometimes the check you want is simple but the answer is not in the right form. The bot returns a JSON object and you only care about one field; or case is getting in the way. A transform is a small JavaScript expression, or a file://, that reshapes the output first. Where you put it decides how far it reaches.

A transform on one assertion

Put transform inside an assertion and it rewrites the output for that check only. Here the classifier returns a JSON object, and the transform parses it and hands the category field to equals, which then compares a plain string.

providers:
  - file://classifier.py
prompts:
  - "{{question}}"
tests:
  - vars: { question: "where is a17" }
    assert:
      - type: equals
        value: delivery
        transform: JSON.parse(output).category

# read out.json: each answer, its checks, and any metric scores
import json

for r in json.load(open("out.json"))["results"]["results"]:
    print(r["response"]["output"], "->", "PASS" if r["success"] else "FAIL")

    for c in r["gradingResult"]["componentResults"]:
        print("  ", "pass" if c["pass"] else "FAIL", c["assertion"]["type"], "-", c["reason"])

    if r["namedScores"]:
        scores = {k: round(v, 3) for k, v in r["namedScores"].items()}
        print("   metrics", scores, "| test score", round(r["score"], 3))

import json

# classify the message and answer in JSON
CATEGORY = {"refund": "billing", "a17": "delivery", "broken": "technical"}

def call_api(prompt, options, context):
    asked = prompt.lower()
    for key, name in CATEGORY.items():
        if key in asked:
            return {"output": json.dumps({"category": name, "priority": 2})}
    return {"output": json.dumps({"category": "other", "priority": 3})}

# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):
    asked = prompt.lower()                      # the rendered prompt is the question
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": line}
    return {"output": "I could not find that in the handbook."}

The stored answer is still the full JSON; only the equals check saw the extracted field.

promptfoo eval -c pick.yaml -o out.json --no-table >/dev/null 2>&1
python3 show.py

{"category": "delivery", "priority": 2} -> PASS
   pass equals - Assertion passed

A transform for every check in a test

Put the transform under defaultTest.options.transform and it rewrites the answer that every assertion in the run sees, and the answer that is stored. This one lower-cases the text, so checks written in lower case match whatever case the bot used.

providers:
  - file://support_bot.py
prompts:
  - "{{question}}"
defaultTest:
  options:
    transform: output.toLowerCase()
tests:
  - vars: { question: "Where is order A17?" }
    assert:
      - type: contains
        value: "3 march"
      - type: starts-with
        value: "order"

promptfoo eval -c lower.yaml -o out.json --no-table >/dev/null 2>&1
python3 show.py

order a17 shipped on 3 march by courier. -> PASS
   pass contains - Assertion passed
   pass starts-with - Assertion passed

Where a transform can live

Where a transform fits

• A JSON reply, where one field is parsed out and checked while the rest is ignored.
• Case or whitespace noise, smoothed once so every check reads clean text.
• A reusable reshape, kept in a file:// and pointed at from the assertions that need it.

Related

• Previous: A JavaScript assertion
• Next: threshold, weight and named metrics
• Reference: Promptfoo configuration reference