Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

llm-rubric: the check that needs a model

llm-rubric is the first assertion in this course that cannot decide on its own: it hands the answer and a plain-language rubric to a model and asks whether the rubric holds.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

The checks so far all decided in code. contains, equals and regex matched text, is-json: checking the shape of an answer read structure, and A Python assertion: a check that is a function ran a function. Some questions have no code test. Whether an answer says the order shipped in March, or reads as polite, is a judgement. llm-rubric asks a model to make it.

Writing an llm-rubric assertion

A rubric is one sentence stating what a passing answer must do. The grader replies with a pass, a score and a reason. This suite checks one grounded answer against one rubric.

yaml
providers:
  - echo
prompts:
  - "{{answer}}"
tests:
  - vars:
      answer: Order A17 shipped on 3 March by courier.
    assert:
      - type: llm-rubric
        value: the answer says the order shipped in March

Running it with no grader set

Nothing in that config names a model. When an llm-rubric runs, promptfoo has to pick a grading provider, and it chooses one from whatever model credentials sit in the environment. With no key present, the run errors before it can grade.

bash
# no grader wired, and no model key in the environment
promptfoo eval --no-table

The error reports that an API key is not set and names a provider to set it for. Which provider it names depends on the environment: it defaults to one vendor on a machine with one set of credentials and to another elsewhere, so there is no fixed message to capture or test against. The point stands on its own: an llm-rubric needs a model, and a model needs a key.

Watch out. The default grading provider is read from the environment, so the no-key error names a different vendor on a different machine. Never write a test or a lesson that depends on its exact words. The rest of this part removes the key requirement entirely by supplying the grader yourself.

Where you use llm-rubric

  • A quality a regex cannot state, such as whether an answer is polite, on-topic or complete.
  • A grounded-answer check, where the rubric names a fact the answer must carry, like a shipping month.
  • A first pass over free text, flagging replies for review rather than matching them exactly.
Try it yourself
  • Change the rubric to the answer is polite and note that without a grader it still cannot run.
  • Add a second test with a different answer and confirm both error for the same missing key.
  • Read the error log path promptfoo prints and open the file it points to.

You understood something today that you didn't yesterday.