What are the main types of evals, and when do you use each?
30-second answerSay your answer out loud first, then reveal.
| Grader | Strengths | Weaknesses | Use for |
|---|---|---|---|
| Code-based | Fast, cheap, deterministic | Only for checkable properties | Format, JSON validity, required fields, numeric correctness, SQL execution results, tests passing |
| LLM-as-judge | Scales, handles nuance | Biases, needs calibration, costs tokens | Faithfulness, helpfulness, tone, policy adherence |
| Human | Most reliable for subjective / expert tasks | Slow, expensive, inconsistent without guidelines | Calibrating judges, high-stakes domains, launch reviews |
Rule of thumb: use code checks wherever possible, LLM judges where necessary, and humans to validate the judges and audit samples.
Related
You understood something today that you didn't yesterday.