When the judge is wrong

A judge is a program with an opinion. Before you let one gate a build, you have to know how often it is wrong, and in which direction.

Here is a rule this course's judge cannot answer, asked anyway.

yaml

providers:
  - file://support_bot.py
prompts:
  - "{{question}}"
defaultTest:
  options:
    provider: file://pretend_promptfoo.py
tests:
  - vars:
      question: How long does a refund take?
    assert:
      - type: llm-rubric
        value: the answer is polite

Example

promptfoo eval

The answer is Refunds take five working days. and the rule asked whether it is polite. The judge compared the word polite against the answer, found nothing, and failed it. The answer is not impolite; the judge simply has no way to tell.

A model judge would have an opinion here, and it would be a better one. It would also be an opinion, arrived at differently on Tuesday than on Monday if the temperature is not zero, and differently again when the model is upgraded under you.

Two ways to be wrong

A judge that passes bad answers is the dangerous one. Your suite goes green, the checks that were supposed to catch a regression report nothing, and you find out from a customer. The stub in lesson 12 was this failure in its purest form.

A judge that fails good answers is the annoying one. The build breaks on something that was fine, somebody re-runs it, and after the third time the check gets deleted or the threshold gets lowered until it never fires.

Calibrating one

The fix is to test the judge the way you test the bot. Write a handful of answers you have already decided about, half of them good and half bad, and run the rubric over them. If the judge agrees with you on all of them, it is worth using on answers you have not seen.

Calibrating a judge

Collect answers you have judged. Ten is enough. Real answers from a real run, not invented ones.Step 1 of 5

Rewrite the rule before you blame the judge. The answer is polite is a rule two people would apply differently, so a model will too. The answer thanks the customer and does not blame them is a rule anything can check. Most judge disagreements are rubric problems wearing a disguise.

Keep the deterministic checks

The checks from part 3 cost nothing, never change their mind, and never need calibrating. Every fact that can be checked with contains or a schema should be, and the judge should be left for the things that genuinely need an opinion. A suite that is all rubrics is slow, expensive and wobbly.

Try it yourself

Rewrite the polite rule as something with words in it, like the answer gives a number of days, and watch it pass.
Write five answers you would accept and five you would not, and run the same rubric over all ten.
Set the judge's passing bar to 0.9 and count how many good answers it now rejects.

Previousfactuality and g-eval: two more formats

Every expert started right here.