What are the main types of evals, and when do you use each?

30-second answerSay your answer out loud first, then reveal.

By grader: code-based checks (deterministic assertions: format, schema, exact match, tests), LLM-as-judge (scales nuanced judgements against a rubric), and human evaluation (the gold standard for subjective or high-stakes quality, but slow and costly). By timing: offline evals on fixed datasets before release, and online evals on live traffic (sampled judging, user feedback, A/B tests). By scope: component-level (retrieval, one step) vs end-to-end (the whole task).

Each grader type (code-based checks, LLM-as-judge, human review) can be used both offline on golden datasets and online on live traffic.

Grader	Strengths	Weaknesses	Use for
Code-based	Fast, cheap, deterministic	Only for checkable properties	Format, JSON validity, required fields, numeric correctness, SQL execution results, tests passing
LLM-as-judge	Scales, handles nuance	Biases, needs calibration, costs tokens	Faithfulness, helpfulness, tone, policy adherence
Human	Most reliable for subjective / expert tasks	Slow, expensive, inconsistent without guidelines	Calibrating judges, high-stakes domains, launch reviews

Rule of thumb: use code checks wherever possible, LLM judges where necessary, and humans to validate the judges and audit samples.

PreviousWhat are evals, and why do LLM applications need them more than traditional software does?

You understood something today that you didn't yesterday.

What are the main types of evals, and when do you use each?

Related