Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

is-json: checking the shape of an answer

is-json checks that an answer is valid JSON, and, when you give it a schema in value, that the JSON has the fields and types you asked for.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

A bot that is meant to return structured data has a second thing to get right beyond the words: the shape. A dropped brace or a missing field breaks whatever reads the answer next. is-json checks the shape with no model, either on its own or against a JSON schema.

A bot that answers in JSON

This classifier sorts a message into a category and a priority and returns them as a JSON object. It is the same kind of plain Python provider as the support bot, but its output is data, not a sentence.

python
import json

# classify the message and answer in JSON
CATEGORY = {"refund": "billing", "a17": "delivery", "broken": "technical"}

def call_api(prompt, options, context):
    asked = prompt.lower()
    for key, name in CATEGORY.items():
        if key in asked:
            return {"output": json.dumps({"category": name, "priority": 2})}
    return {"output": json.dumps({"category": "other", "priority": 3})}

Valid JSON, then a schema

The first test asks only that the answer parse as JSON. The next adds a schema in value: an object that must have category and priority. A second assertion on the same answer uses a stricter schema that also demands an eta field the classifier never sends, so it fails while the looser one passes.

yaml
providers:
  - file://classifier.py
prompts:
  - "{{question}}"
tests:
  - vars: { question: "I need a refund" }
    assert:
      - type: is-json
  - vars: { question: "where is a17" }
    assert:
      - type: is-json
        value:
          type: object
          required: [category, priority]
      - type: is-json
        value:
          type: object
          required: [category, priority, eta]
Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports these files. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep them in the same folder.
View the code here
show.py
# read out.json: each answer, its checks, and any metric scores
import json

for r in json.load(open("out.json"))["results"]["results"]:
    print(r["response"]["output"], "->", "PASS" if r["success"] else "FAIL")

    for c in r["gradingResult"]["componentResults"]:
        print("  ", "pass" if c["pass"] else "FAIL", c["assertion"]["type"], "-", c["reason"])

    if r["namedScores"]:
        scores = {k: round(v, 3) for k, v in r["namedScores"].items()}
        print("   metrics", scores, "| test score", round(r["score"], 3))
support_bot.py
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):
    asked = prompt.lower()                      # the rendered prompt is the question
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": line}
    return {"output": "I could not find that in the handbook."}

Reuse the reader from the previous lesson to see each shape verdict.

ExampleValid JSON passes; the schema catches the missing field
promptfoo eval -c jsonbot.yaml -o out.json --no-table >/dev/null 2>&1
python3 show.py

A malformed answer fails

Point is-json at the prose support bot instead. Its answer is a plain sentence, which is not JSON, so the check fails and names why.

yaml
providers:
  - file://support_bot.py
prompts:
  - "{{question}}"
tests:
  - vars: { question: "Where is order A17?" }
    assert:
      - type: is-json
ExampleA sentence is not JSON
promptfoo eval -c malformed.yaml -o out.json --no-table >/dev/null 2>&1
python3 show.py

What is-json checks, with and without a schema

valueWhat is-json requires
NoneThe output parses as JSON
A JSON schemaThe output parses, and matches the schema's fields and types

Where a shape check fits

  • A bot that returns JSON, where a dropped brace should fail the test, not the parser downstream.
  • A required field, named in a schema so an answer that forgets it is caught here.
  • A tool-call payload, checked for shape before any check on the values inside it.
Watch out. is-json with no schema only asks that the text parse, so [] and "hi" both pass. When the answer must be an object with set fields, pass a schema in value; the empty check is a weak guard on its own.
Try it yourself
  • Add a schema that requires priority to be a string and watch the classifier's number fail it.
  • Feed is-json the string [] with no schema and confirm it passes.
  • Give the malformed test a schema and note the reason changes from invalid JSON to a schema error.

You understood something today that you didn't yesterday.