Why is building an eval set with the customer so important, and how do you do it?

30-second answerSay your answer out loud first, then reveal.

The eval set turns subjective opinions ("it seems good") into objective progress, aligns everyone on what "correct" means, catches regressions, and is the evidence for go/no-go decisions. Build it with the customer's subject-matter experts: collect real, representative examples (including hard and edge cases), define grading rubrics together, have SMEs label or verify reference outputs, measure agreement between SMEs, and version it as the project evolves.

Process

Sample real data: 100–300 cases stratified by type, difficulty, language and source. Include known-hard cases and examples the AI should refuse or escalate.
Define the rubric with SMEs: e.g. for summaries: correct facts, no critical omissions, correct recommendation, appropriate tone. Make the criteria binary or 1–3 scales with examples.
Label: SMEs write or approve reference outputs; for extraction, field-level ground truth.
Calibrate: two SMEs label an overlapping subset. If they disagree often, the task definition is unclear, so fix the definition first.
Automate scoring where possible: exact matches for fields, an LLM judge calibrated against SME grades for free text.
Version and grow: add production failures; keep a held-out portion to avoid overfitting prompts.

Why customers value it: it gives them confidence, it's an asset they keep (useful for vendor comparisons and future upgrades), and it gives SMEs ownership of quality.

Practical tips: SME time is scarce, so make labelling easy (a simple UI or spreadsheet), batch sessions, and show them how their labels improved the system.

PreviousHow do you define success metrics with a customer for an AI project?

Little by little, you're building something great.

Why is building an eval set with the customer so important, and how do you do it?

Process

Related