1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
51 small wins to finish your pathNext question →
How do you design a good LLM-as-judge prompt?
30-second answerSay your answer out loud first, then reveal.
Template structure
You are evaluating whether a support answer is GROUNDED in the provided documents.
PASS if every factual claim in the answer is directly supported by the documents.
FAIL if any claim is not supported, contradicts the documents, or adds details
(numbers, dates, policy terms) not present in them.
Ignore style and tone.
Examples:
<example verdict="PASS">...</example>
<example verdict="FAIL">... (adds "within 7 days", docs say 10 days)</example>
<documents>...</documents>
<question>...</question>
<answer>...</answer>
Return JSON: {"reasoning": "<2-3 sentences>", "verdict": "PASS" | "FAIL"}Design guidelines
- Binary verdicts are more reliable and actionable than 1–10 scores (what's the difference between a 6 and a 7?).
- Specific criteria from error analysis ("asked for order ID before checking status") beat generic ones ("helpfulness").
- Few-shot examples of borderline cases, labelled by experts.
- Reasoning before the verdict improves accuracy and makes debugging easier.
- A strong model as judge for nuanced criteria; small models for simple checks.
- Control biases: randomise order in pairwise comparisons; tell the judge not to favour length.
- Version the judge prompt like any other prompt, and re-validate it after changes.
Common mistakes
- One judge prompt scoring "overall quality 1–10".
- Not giving the judge the source documents, then asking it about faithfulness.
Related
Little by little, you're building something great.