What is Goodhart's law in the context of evals, and how do you avoid overfitting to your eval set?

30-second answerSay your answer out loud first, then reveal.

"When a measure becomes a target, it ceases to be a good measure." Teams iterating heavily on prompts against a fixed eval set can improve the score without improving real quality: special-casing examples, exploiting judge biases (verbosity), or narrowing behaviour in ways the set doesn't penalise. Avoid it with held-out test sets used rarely, regular refreshes from production, multiple complementary metrics, judge re-validation, human spot checks, and online A/B validation.

How overfitting happens

Adding few-shot examples copied from eval cases.
Prompt rules that patch individual failures ("if the user asks about X, say Y").
Optimising toward a judge's quirks (e.g. longer answers score higher).
Automated prompt optimisers (e.g. DSPy-style) tuning directly on the test set.

Safeguards

Safeguard	How
Train / dev / test splits	Iterate on dev; evaluate on test only at milestones
Fresh data	Monthly additions from production; rotate examples
Multiple metrics	Quality + conciseness + refusal rate + cost
Judge diversity and calibration	Re-validate judges; use different judge models occasionally
Human audits	Periodic blind human review of random production samples
Online validation	A/B tests on real user outcomes
Generalisation checks	Paraphrased versions of eval inputs should score similarly

Interview line. "My eval set is a sample of reality, not reality. I protect a held-out set and keep refreshing from production."

PreviousLeadership asks you to cut the hallucination rate of a customer-facing assistant in half within a quarter. How do you run the program?

Little by little, you're building something great.

What is Goodhart's law in the context of evals, and how do you avoid overfitting to your eval set?

How overfitting happens

Safeguards

Related