llm-rubric: the check that needs a model

Every check so far compares strings. Some things worth testing cannot be written that way: whether an answer is polite, whether it refused when it should have, whether it stayed on the subject. For those, promptfoo asks a model.

These are the model-graded assertions, and llm-rubric is the one they are all built on. You write the rule in a sentence, promptfoo sends the answer and the sentence to a grading model, and reads back a verdict.

assert:
  - type: llm-rubric
    value: the answer says the order shipped in March

What happens with no key

Run that on a fresh machine and it fails, but not because the answer was wrong. Promptfoo picks a default grading provider from whatever credentials it finds in your environment, and with none it has nothing to ask. The error names whichever provider it tried, so the exact wording depends on what else you have installed.

This is the wall every other promptfoo tutorial hits at exactly this point, and where they tell you to go and get an API key.

The grader is just a provider

Here is the way through. The grading model is set with provider under options, and it takes the same values as the providers list at the top of the file. Anything that can answer a prompt can grade one, including a Python file.

defaultTest:
  options:
    provider: file://pretend_promptfoo.py

defaultTest applies to every test in the suite, so one block covers the whole file. The same key on a single test covers that test, and provider on an individual assertion covers that assertion, which is how you use a cheap grader everywhere and an expensive one on the two checks that need it.

A judge that agrees with everything

The smallest thing that satisfies the contract is a file that returns a verdict without looking at anything.

import json


def call_api(prompt, options, context):
    """A judge that agrees with everything. Useless, but it runs."""
    return {"output": json.dumps({"pass": True, "score": 1, "reason": "looks fine"})}

The same two questions as before, each with a rule written as a sentence.

tests:
  - vars:
      question: Where is order A17?
    assert:
      - type: llm-rubric
        value: the answer says the order shipped in March
  - vars:
      question: Where is order B99?
    assert:
      - type: llm-rubric
        value: the answer says the order shipped in March

promptfoo eval

Both tests pass, and the second one should not have. The bot answered I could not find that in the handbook and the rule asked for a shipping date in March. The judge never looked.

The next lesson makes it look.