How do you make eval results statistically trustworthy?

30-second answerSay your answer out loud first, then reveal.

Account for sample size and randomness. Report confidence intervals (pass rates are binomial: SE ≈ √(p(1−p)/n)), run multiple trials per example when outputs are non-deterministic, use paired comparisons on the same examples when comparing versions (e.g. a paired bootstrap or McNemar's test), size datasets so the expected effect is detectable, and avoid over-interpreting small differences or cherry-picked slices.

Confidence interval intuition

text

95% ≈ ±1.96 × SE

n examples	Measured 80% pass	95% CI (approx.)
50	80%	±11 points
200	80%	±5.5 points
1,000	80%	±2.5 points

"Prompt B scored 83% vs A's 80% on 100 examples" is likely noise.

Techniques

Paired comparison: evaluate both versions on the same examples and look at the examples where they differ. That's far more sensitive than comparing two independent averages.
Bootstrap confidence intervals for complex metrics.
Multiple runs: for stochastic systems, run k times. Report mean ± spread, or pass^k (all runs pass) for reliability.
Judge noise: include judge error in the uncertainty (Q17).
Slices: a significant overall change may hide regressions in a small slice; but slicing into many groups risks false discoveries, so pre-define the important slices.
Practical significance: a statistically significant 0.5% gain may not justify a 30% cost increase.

Interview signal. Spontaneously mentioning confidence intervals and paired comparisons is rare and impressive.

PreviousPointwise vs pairwise evaluation: when do you use each? How do arena-style rankings work?

This is what real progress feels like.

How do you make eval results statistically trustworthy?

Confidence interval intuition

Techniques

Related