is-json: checking the shape of an answer
is-json checks that an answer is valid JSON, and, when you give it a schema in value, that the JSON has the fields and types you asked for.
Last updated: 15 Oct, 2026 · Promptfoo 0.124.1
A bot that is meant to return structured data has a second thing to get right beyond the words: the shape. A dropped brace or a missing field breaks whatever reads the answer next. is-json checks the shape with no model, either on its own or against a JSON schema.
A bot that answers in JSON
This classifier sorts a message into a category and a priority and returns them as a JSON object. It is the same kind of plain Python provider as the support bot, but its output is data, not a sentence.
import json
# classify the message and answer in JSON
CATEGORY = {"refund": "billing", "a17": "delivery", "broken": "technical"}
def call_api(prompt, options, context):
asked = prompt.lower()
for key, name in CATEGORY.items():
if key in asked:
return {"output": json.dumps({"category": name, "priority": 2})}
return {"output": json.dumps({"category": "other", "priority": 3})}
Valid JSON, then a schema
The first test asks only that the answer parse as JSON. The next adds a schema in value: an object that must have category and priority. A second assertion on the same answer uses a stricter schema that also demands an eta field the classifier never sends, so it fails while the looser one passes.
providers:
- file://classifier.py
prompts:
- "{{question}}"
tests:
- vars: { question: "I need a refund" }
assert:
- type: is-json
- vars: { question: "where is a17" }
assert:
- type: is-json
value:
type: object
required: [category, priority]
- type: is-json
value:
type: object
required: [category, priority, eta]
View the code here
# read out.json: each answer, its checks, and any metric scores
import json
for r in json.load(open("out.json"))["results"]["results"]:
print(r["response"]["output"], "->", "PASS" if r["success"] else "FAIL")
for c in r["gradingResult"]["componentResults"]:
print(" ", "pass" if c["pass"] else "FAIL", c["assertion"]["type"], "-", c["reason"])
if r["namedScores"]:
scores = {k: round(v, 3) for k, v in r["namedScores"].items()}
print(" metrics", scores, "| test score", round(r["score"], 3))
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
"a17": "Order A17 shipped on 3 March by courier.",
"refund": "Refunds take five working days.",
}
def call_api(prompt, options, context):
asked = prompt.lower() # the rendered prompt is the question
for key, line in HANDBOOK.items():
if key in asked:
return {"output": line}
return {"output": "I could not find that in the handbook."}
Reuse the reader from the previous lesson to see each shape verdict.
promptfoo eval -c jsonbot.yaml -o out.json --no-table >/dev/null 2>&1
python3 show.py{"category": "billing", "priority": 2} -> PASS
pass is-json - Assertion passed
{"category": "delivery", "priority": 2} -> FAIL
pass is-json - Assertion passed
FAIL is-json - JSON does not conform to the provided schema. Errors: data must have required property 'eta'A malformed answer fails
Point is-json at the prose support bot instead. Its answer is a plain sentence, which is not JSON, so the check fails and names why.
providers:
- file://support_bot.py
prompts:
- "{{question}}"
tests:
- vars: { question: "Where is order A17?" }
assert:
- type: is-json
promptfoo eval -c malformed.yaml -o out.json --no-table >/dev/null 2>&1
python3 show.pyOrder A17 shipped on 3 March by courier. -> FAIL FAIL is-json - Expected output to be valid JSON
What is-json checks, with and without a schema
| value | What is-json requires |
|---|---|
| None | The output parses as JSON |
| A JSON schema | The output parses, and matches the schema's fields and types |
Where a shape check fits
- A bot that returns JSON, where a dropped brace should fail the test, not the parser downstream.
- A required field, named in a schema so an answer that forgets it is caught here.
- A tool-call payload, checked for shape before any check on the values inside it.
is-json with no schema only asks that the text parse, so [] and "hi" both pass. When the answer must be an object with set fields, pass a schema in value; the empty check is a weak guard on its own.Related
- Previous: Deterministic checks: contains, equals, regex
- Next: A Python assertion: a check that is a function
- Reference: Promptfoo deterministic assertions
- Add a schema that requires
priorityto be a string and watch the classifier's number fail it. - Feed
is-jsonthe string[]with no schema and confirm it passes. - Give the malformed test a schema and note the reason changes from invalid JSON to a schema error.
You understood something today that you didn't yesterday.