Correct your judge's score with TPR and TNR

Lesson 2.4 · 4 min read · by Vinay C

We'll cover the following

A judge makes mistakes in both directions, so its raw pass rate is not the true pass rate. Measure its error rates against human labels, then correct the score.

The two numbers

  • TPR (true positive rate): of the answers humans call good, how many does the judge pass?
  • TNR (true negative rate): of the answers humans call bad, how many does the judge fail?
  • Report both. One "85% accurate" number hides which way the judge errs: a judge that passes almost everything can still look accurate.

Step 1: build a confusion matrix

Label about 100 examples by hand and run the judge on the same ones.

Judge: passJudge: fail
Human: good455
Human: bad1040

TPR = 45/50 = 90%. TNR = 40/50 = 80%.

Step 2: correct the observed score

true_rate = (observed + TNR - 1) / (TPR + TNR - 1)

observed 0.70, TPR 0.90, TNR 0.80  ->  0.50 / 0.70 = ~71%
observed 0.70, TPR 0.95, TNR 0.60  ->  0.30 / 0.55 = ~55%

The same "70%" is a 71% system with a strict judge and a 55% system with a lenient one. This is the Rogan-Gladen correction, borrowed from epidemiology for imperfect medical tests.

Watch out for

  • If TPR + TNR - 1 is near 0, the judge is barely better than a coin flip. You cannot correct it, so fix the judge instead.
  • Your 100 labels have error bars too. Bootstrap the labels and the eval set together.
  • Comparing v1 and v2 is only fair if the judge's error rates stay the same. If v2 writes longer, more confident answers, the judge's TNR can drop on v2 and part of the "gain" is leniency.
  • Re-calibrate when you change the judge model, the judge prompt, or the style of the system's answers.
  • Low TNR means the judge misses subtle failures. Add examples of those failures to the judge prompt. Changing the system will not help.
  • Keep a small human-labeled set as a permanent test set for the judge itself.
Check yourselfYour judge says 70% pass. Calibration shows TPR 0.95 and TNR 0.60. What is the corrected pass rate, and what does it tell you?

(0.70 + 0.60 - 1) / (0.95 + 0.60 - 1) = 0.30 / 0.55, about 55%. The judge is lenient (low TNR): it passes too many bad answers, so the raw 70% overstates quality.

Summary: say it in the interview

  • "I don't trust the judge's raw score. I calibrate it against human labels, report TPR and TNR, and correct the pass rate for the judge's error."