RAGA Framework, Part 6: Who Evaluates the Evaluator? The Limits of LLM-Based Metrics

Part 6 of a series on evaluating LLM apps with Ragas.

The Limits of LLM-Based Metrics - by Vinay C

Everything in this series so far rests on a quiet assumption. When Ragas tells you faithfulness is 0.83, an LLM produced that number by reading the answer and the context and making a judgment. It's an LLM grading an LLM.

If the grader is biased or noisy, every decision built on its scores inherits that flaw.

This is the article I'd least want you to skip. Being able to say "here's why I don't fully trust my own metrics, and here's what I do about it" is what separates someone who runs evals from someone who understands them.

Where LLM judges go wrong

None of these are reasons to abandon LLM-based metrics. They're reasons to hold the scores loosely and validate them.

Verbosity bias. LLM judges tend to reward longer, more detailed answers, even when the extra detail adds nothing or is subtly wrong. A concise correct answer can score below a padded one.

Prompt sensitivity. The judge is driven by a prompt. Small wording changes in that prompt can shift scores. Ragas ships sensible defaults, but the moment you customize a metric's prompt, you've changed the ruler.

Non-determinism. Run the same evaluation twice and you may get slightly different scores. This matters when you compare runs: a big gap like 0.61 versus 0.79 is almost certainly a real difference, but a small one like 0.81 versus 0.83 could just be that random wobble, not an actual improvement. Before you celebrate a tiny bump, you need to know how much your scores move on their own.

Self-preference. A judge model can favor text produced by itself or by models in its own family. If your generator and your judge are the same model, be suspicious of flattering scores.

Domain-knowledge gaps. A general-purpose judge doesn't know your company's policies, your legal constraints, or your product's edge cases. It can rate an answer faithful to the retrieved text while missing that the text itself is out of date or misapplied.

Judge disagreement. Two different judge models will disagree with each other, sometimes a lot. There's no single "true" LLM score.

Threshold fragility. Teams love a rule like "ship if faithfulness > 0.8." But 0.8 is arbitrary unless you've checked what scores near that line actually look like. The threshold is a decision you have to earn, not a default.

Cost and latency. LLM-judged metrics mean extra model calls per sample. A large eval set can get slow and expensive, which quietly pushes people toward evaluating less often, exactly the wrong direction.

Humans disagree too, and that's the point

It's tempting to treat human ratings as ground truth. They aren't clean either. Ask two people to rate the same thirty answers and they'll disagree on a meaningful chunk, especially on fuzzy qualities like relevance.

That's not a reason to dismiss human review. It's a reason to treat evaluation as calibration between imperfect raters rather than a hunt for one true score.

So the question isn't "is the LLM judge perfect?" It's "does the LLM judge agree with careful humans often enough to trust its direction?"

A validation exercise worth an afternoon

Here's a concrete way to find out whether your metrics track reality, using our policy assistant. This is the single highest-value thing in the whole series if you're shipping something people rely on.

1. Sample 30 to 50 responses from your system, spanning good and bad.
2. Have two humans rate them independently on the axis you care about, say faithfulness: is every claim supported by the retrieved policy text? Use a simple scale.
3. Run the Ragas metric on the same set.
4. Compare. Where do human and Ragas scores agree, and where do they diverge?
5. Inspect the biggest mismatches by hand. This is where the learning is. Read the cases where Ragas said 0.9 and your humans said "this is wrong."
6. Adjust. Depending on what you find, refine the metric prompt, change the threshold, or switch judge models. Ragas supports customizing metric prompts, and even aligning a metric against your own labeled examples so it matches your judgment more closely.

When I did this, the mismatches were the education. Ragas rated several answers highly faithful because every sentence traced back to the retrieved text, but the retrieved text was an outdated policy version. The metric was doing its job correctly, faithfulness only checks the answer against the retrieved context. My mental model was wrong: I'd been reading "faithful" as "correct," and they are not the same thing. That single insight changed how I read every score afterward.

Practical guardrails

• Don't over-read small gaps. Treat sub-0.02 differences as noise unless you've measured run-to-run variance and know better.
• Prefer non-LLM metrics where they fit. For retrieval you can often use ID-based or string-similarity precision, which are deterministic and free. Use the LLM judge where nothing cheaper captures the quality.
• Keep judge and generator different when you can, to dodge self-preference.
• Re-validate periodically. A judge you calibrated six months and two model upgrades ago may have drifted.
• Report metrics with humility. "Faithfulness is 0.83 on our judge, validated against human ratings with reasonable agreement" is honest. "Faithfulness is 0.83" alone oversells it.

Interview angle

The question: "Can you trust the scores from an LLM-based metric?"

The answer:

• Name the failure modes. LLM judges have verbosity bias, prompt sensitivity, and self-preference, so the scores aren't ground truth.
• Validate against humans. Check the metric against a small human-labeled set before you rely on it.
• Be careful with thresholds. Don't treat a tiny score bump as a real win until you know your run-to-run variance.

A great line to land: "I treat automated evals as a scaling mechanism for human judgment, not a replacement for it."

Expect follow-ups on how you'd detect judge drift or reduce evaluation cost.

The takeaway

Automated evaluation lets you scale judgment to thousands of examples. It does not make that judgment infallible. Validate your metrics against human ratings on a small sample, inspect the disagreements, and read every score knowing what it does and doesn't measure.

So far the whole series has been about single-response systems: one question, one retrieval, one answer. Modern applications increasingly take many steps and call tools. Evaluating those is a different problem, and it's where the next article goes.