Design a human review and annotation operation that supports evals at scale (thousands of items per week).

30-second answerSay your answer out loud first, then reveal.

Treat human review as a production system. Write clear guidelines with examples, recruit and qualify reviewers (domain experts for specialised tasks), use a tooling pipeline (task queues, a review UI with context, structured labels and reason codes), add quality control (gold questions, overlap for agreement, adjudication, reviewer performance tracking), use smart sampling (uncertain, high-risk and random items), protect privacy, and feed labels into eval datasets and judge calibration.

Items are prioritised and sampled, mixed with gold items into task queues by skill, reviewed in a review UI, checked by quality control, and turned into approved labels for eval datasets and judge calibration plus reviewer metrics that drive training and guideline updates.

Key elements

Guidelines: definitions, decision trees, many examples (especially borderline ones), versioned; updated when disagreements reveal ambiguity.
Reviewer management: qualification tests, ongoing gold-question accuracy, calibration sessions, wellbeing support (for reviewers exposed to harmful content).
Throughput and cost: estimate minutes per item × volume; for complex items, use LLM pre-labelling with human verification (faster, but watch anchoring bias).
Agreement metrics: Cohen's / Krippendorff's alpha; disagreement queues to senior adjudicators.
Privacy: PII redaction where possible, access controls, secure environments, NDAs.
Closing the loop: label distributions and new failure categories feed back into error analysis and product teams.

PreviousDesign an action-guardrail layer (policy engine) for agents that operate on business systems.

Slow is fine. Stopping is the only problem.

Design a human review and annotation operation that supports evals at scale (thousands of items per week).

Key elements

Related