Your LLM judge disagrees with domain experts 30% of the time. What do you do?

30-second answerSay your answer out loud first, then reveal.

Analyse the disagreements before changing anything. Check whether the experts agree with each other (if not, the criterion itself is ambiguous), whether the judge lacks context the experts use (policies, domain knowledge, source documents), whether the rubric is too vague or bundles several criteria, and whether particular case types drive the disagreement. Then refine the definitions, split criteria, add examples and context, try a stronger judge model, and re-measure on a held-out set. Use humans for the criteria the judge can't do reliably.

Diagnosis steps

Inter-expert agreement: if two experts agree only 75% of the time, the judge can't do much better. Clarify the criterion with the experts first.
Categorise the disagreements:

Pattern	Fix
Judge too lenient on subtle factual errors	Add explicit failure conditions and examples
Judge lacks domain knowledge (e.g. drug interactions)	Provide reference material, or use expert-only review for this criterion
Rubric bundles "correct + complete + polite"	Split into separate binary judges
Disagreement concentrated in one category	Category-specific instructions or examples
Judge inconsistent across runs	Temperature 0; majority vote of 3

3. Iterate on a dev split; confirm on a test split (avoid tuning to the test set).

4. Decide the usage policy: if agreement stays low, use the judge only for triage (flag likely failures for human review) rather than as the final metric.

Interview signal. Treating the judge as a model to validate, plus recognising the human-agreement ceiling.

PreviousYour offline eval scores improved after a change, but user complaints increased. What's going on?

You understood something today that you didn't yesterday.

Your LLM judge disagrees with domain experts 30% of the time. What do you do?

Diagnosis steps

Related