Is your eval gain real or noise?
We'll cover the following
An eval score is a measurement with error bars. v1 scores 78% and v2 scores 83% on 100 questions. That 5-point gain is usually inside the noise.
The error bar
SE = sqrt(p(1-p)/n) = sqrt(0.8 x 0.2 / 100) = ~4%
95% CI = about +/- 8 pointsv1 sits somewhere in 70-86% and v2 in 75-91%. Those ranges overlap heavily.
Three habits
- Compare paired, not separately. Run both versions on the same questions and count only the flips (for example 12 where v2 is right and v1 wrong, 7 the other way). Run a sign test or McNemar's test on those. Questions both versions get right or wrong carry no information, and pairing removes them.
- Find your noise floor. Even at temperature 0, judges, tool calls and batching add randomness. Run the same version 3 to 5 times. If the score moves 3 points, any gain under 3 points is invisible.
- For agents, separate
pass@kfrompass^k.pass@kmeans at least one success in k tries (can it ever do this?).pass^kmeans success on all k tries (what users feel). A 90% per-try agent haspass^5of about 0.9^5 = 59%.
How many questions you need
Unpaired, at about 95% confidence: n ~ 8 x p(1-p) / d^2 per version, where d is the gap you want to detect.
| Gap to detect | Questions needed |
|---|---|
| 10 points | ~150 |
| 5 points | ~600 |
| 2 points | ~4,000 |
Pairing cuts these numbers a lot. With a 50-question set you can only trust big wins.
Check yourselfv2 beats v1 by 5 points on a 100-question set. Do you ship it?
Not yet. The 95% CI is about +/-8 points per version, so the gain is within noise. Compare paired on the same questions, read the flips, repeat runs to measure the noise floor, and add questions if the gain is still unclear.
Summary: say it in the interview
- "A 5-point gain on 100 questions is inside the noise. I'd compare both versions on the same set, look at the questions that flipped, repeat runs to find the noise floor, and ship only if the gain clears it."