Can you trust an LLM judge?
We'll cover the following
LLM-as-judge scales, but it is a noisy instrument. Know how it fails.
How it fails
- Inconsistency: the same input can score differently across runs, and different judge models give different numbers (0.8 from one, 0.5 from another).
- Self-preference: a judge favors answers written by its own model family.
- Verbosity bias: longer answers score higher even when they are not better.
- Position bias: in A-vs-B comparisons it favors whichever comes first.
- Vague scales: "rate 1 to 10" gives mush.
What you do about it
- Temperature 0, a fixed rubric, and average a few runs.
- Binary or 3-point rubrics with clear criteria beat 1-10 scales.
- Pairwise comparisons: run both orders and swap positions.
- Calibrate against humans: label 50 to 100 examples by hand, run the judge on the same ones, and check agreement. Fix the rubric until they agree.
- Use the judge to compare (before vs after a change) with the same judge every time, not to trust absolute scores.
Check yourselfYour judge gives 0.8 and a teammate's judge gives 0.5 on the same output. Who is right?
Neither can be trusted in absolute terms. Scores depend on the judge model and rubric. Compare before vs after with the same judge, and check the judge against human labels.
Summary: say it in the interview
- "I calibrate the judge against a small human-labeled set, then use it for relative comparisons with the same judge model, because the absolute numbers drift between models."