is-json: checking the shape of an answer

A support bot answers in sentences. A classifier answers in JSON, and then the first thing worth testing is not what it said but whether the shape is right at all.

Here is a second provider. Same idea as the support bot, but it returns a category and a priority as JSON, the way an application would when something downstream has to read it.

import json

CATEGORY = {"refund": "billing", "a17": "delivery", "broken": "technical"}

def call_api(prompt, options, context):
    asked = prompt.lower()
    for key, name in CATEGORY.items():
        if key in asked:
            return {"output": json.dumps({"category": name, "priority": 2})}
    return {"output": json.dumps({"category": "other", "priority": 3})}

Three checks, getting stricter

is-json with no value asks only whether the answer parses. Given a value, it checks the parsed answer against a JSON Schema. And javascript runs an expression over the answer for anything a schema cannot say.

assert:
  - type: is-json
  - type: is-json
    value:
      required: [category, priority]
      type: object
      properties:
        category:
          type: string
          enum: [billing, delivery, technical, other]
        priority:
          type: integer

A third check goes below them, for the one thing a schema cannot say: which category this particular question should get.

  - type: javascript
    value: JSON.parse(output).category === 'billing'

promptfoo eval

All three passed. The schema did the real work: it says the answer must be an object, must have both keys, and that category has to be one of four words. A model that invents a fifth category fails here rather than three services downstream.

Why the schema is the check worth writing

Parsing is a low bar. Anything that parses can still be wrong in a way that breaks the next system: a string where a number was expected, a missing field, an enum value nobody planned for. A schema is the cheapest place to say what you actually require, and unlike a sentence-level check it never becomes stale when the wording changes.

Reaching into the answer

When you only care about one field, transform pulls it out and every later check on that test sees the smaller value.

assert:
  - type: is-json
  - type: equals
    value: billing
    transform: JSON.parse(output).category