RAGA Framework, Part 2: Four Metrics That Reveal Why Your RAG System Is Failing

Part 2 of a series on evaluating LLM apps with Ragas.

RAGAs LLM Framework - infographic - by Vinay C

In the last article I argued that "it seems fine" is not a measurement. This one is about the four numbers I reach for first when a RAG system is misbehaving, and how reading them together tells you where the problem lives.

The way I keep them straight is to follow the data through the pipeline, from the source documents to the final answer, and ask one question at each stage:

1. Did we retrieve everything we needed? → context recall
2. Did we rank it right? → context precision
3. Is the model faithfully using it, or hallucinating? → faithfulness
4. Does the answer actually address the question? → response relevancy

Notice the arc: it starts at the user's question (that's what retrieval runs on) and ends back at the user's question (does the answer respond to it). Two metrics are about the retriever, two are about the generator. Keep that flow in your head and you can debug most RAG failures in the right order.

I'll use one running example for the rest of the series: a company policy assistant. Employees ask it things like "how many days of paternity leave do I get?" and it answers from the HR policy documents. It's a good example because the failure modes are the ones you actually hit at work, and because a wrong answer here has real consequences.

Let me start with a single bad interaction and take it apart.

Question: "How much paternity leave do I get, and do I need to take it all at once?"

What the retriever pulled back:

1. A paragraph about maternity leave (12 weeks, paid).
2. A paragraph about the general PTO accrual policy.
3. Two paragraphs of the company's history and mission.

What the assistant answered: "You are entitled to 12 weeks of paid leave, which must be taken in a single continuous block."

That answer is fluent, confident, and wrong. Paternity leave in this company is 4 weeks and can be split. The assistant grabbed the maternity paragraph, treated it as paternity, and invented the "single continuous block" rule. A human skimming it would probably nod. This is exactly the kind of failure vibe checks miss.

Now let's walk the four questions in order.

One thing up front, so the numbers later in the series mean something: each of these scores runs from 0 to 1, higher is better. There is no universal passing grade. A faithfulness of 0.9 might be perfectly fine for an internal FAQ bot and completely unacceptable for anything touching legal, medical, or financial answers. A score only has meaning relative to your own baseline and your own tolerance for being wrong. How to set actual thresholds comes later in the series; for now, read these as "higher is better, and compare against yourself."

1. Did we retrieve everything we needed? — Context recall

Start here, because this is the most fundamental question. If the evidence needed to answer never gets pulled out of your documents, nothing downstream can save you. The model is left guessing or hallucinating.

Context recall asks: of the information required to answer correctly, how much did retrieval actually surface? It's measured against a reference answer (a human-written correct answer), by checking whether each claim in the reference can be attributed to the retrieved context. So recall is one of the metrics that needs you to have prepared reference answers in advance.

Our example fails recall badly. The correct answer needs the paternity paragraph (4 weeks, splittable), and that paragraph never made it into the retrieved set. No ranking trick or prompt tweak can recover it. If it's not retrieved, it's not there.

This is the metric that most often explains a "confidently wrong" system, and it's the one people forget to check first.

2. Did we rank it right? — Context precision

Recall asks whether the right chunks were found at all. Precision asks whether, among what was retrieved, the relevant chunks are ranked above the irrelevant ones. It's computed as a mean of precision@k down the ranked list, so a relevant chunk sitting at position 1 helps your score far more than the same chunk buried at position 4. That weighting matters because most generation prompts lead with the top chunks, and some truncate the tail entirely.

In our example, precision is poor too. There wasn't even a real paternity paragraph in the retrieved set, and the top slots went to maternity leave and company history. The retriever ranked noise highly.

Recall and precision are really two properties of the same retrieval step: recall is "did we get all the right stuff," precision is "is the right stuff on top." I check recall first because completeness is the harder gate, but you read them together.

Ragas offers a few flavors of precision, and knowing the difference is worth interview points:

• LLMContextPrecisionWithReference compares each chunk to a reference answer.
• LLMContextPrecisionWithoutReference compares each chunk to the generated response, for when you have no reference.
• NonLLMContextPrecisionWithReference and IDBasedContextPrecision use string similarity or document IDs instead of an LLM judge, which is cheaper and deterministic.

3. Is the model faithfully using it, or hallucinating? — Faithfulness

Now we cross from the retriever to the generator. Assume retrieval did its job; the question becomes whether the model actually stuck to what it was given.

Faithfulness asks whether the claims in the answer are supported by the retrieved context. Ragas breaks the answer into individual claims and checks each one against the context. It needs no reference answer, it just compares the answer against the chunks that were retrieved.

In our example it's mixed. "12 weeks of paid leave" is at least traceable to the (wrong) maternity paragraph that was retrieved. But "must be taken in a single continuous block" appears nowhere in the context. The model made it up. Faithfulness catches that fabricated claim even though the sentence reads perfectly.

This deserves a pause, because it's a classic interview trap: faithfulness is not correctness. Faithfulness only asks "does the answer match the retrieved text." Whether that text is true, current, or even the right document is a separate question entirely. An answer can be perfectly faithful to a paragraph that is outdated or wrong, and score high. To check whether the answer is actually correct, you compare it against a known-good reference answer, which in Ragas is a different metric (factual correctness, sometimes called answer accuracy). Put simply: faithfulness catches the model inventing things; correctness catches the model confidently repeating a bad source. You need both, and confusing the two is exactly the mistake this series keeps warning about.

4. Does the answer actually address the question? — Response relevancy

The last question closes the loop back to where we started: the user's question.

Response relevancy asks whether the answer actually responds to what was asked, ignoring for a moment whether it's correct. Ragas estimates this by generating questions the answer could be answering and measuring how close they are to the real question. It needs no reference answer, but it does use an embedding model to compare those questions, so it's the one metric here that depends on embeddings rather than just an LLM judge. It's also the only one of the four that never looks at the retrieved context, it compares the answer straight back to the question.

Our answer is reasonably relevant on this axis. The user asked about paternity leave duration and whether it can be split; the answer addresses both, just incorrectly. So relevancy looks decent even though the content is wrong. This is the trap: a high relevancy score tells you the model understood the question, not that it got the answer right.

What each metric needs

Before combining them, it helps to know what data each metric costs you and whether it leans on an LLM judge. The four data fields in play are: user_input (the question), retrieved_contexts (the chunks pulled), response (the answer), and reference (the golden answer, the human-written correct response).

Metric Data it needs Reference answer? LLM judge? Context recall user_input, retrieved_contexts, reference Yes Yes Context precision (with reference) user_input, retrieved_contexts, reference Yes Yes Faithfulness user_input, retrieved_contexts, response No Yes Response relevancy user_input, response No Yes + embeddings

Two things fall out of this table. First, the split you'd expect: the retriever metrics (recall, precision) lean on a reference, the generator metrics (faithfulness, relevancy) don't, because faithfulness checks the answer against the context and relevancy checks it against the question. Response relevancy is the one exception to "just an LLM judge", it also needs an embedding model to compare questions.

Second, the LLM judge isn't always mandatory. Ragas has non-LLM retrieval variants (NonLLMContextPrecisionWithReference, IDBasedContextPrecision, and similar) that use string similarity or document IDs instead of a judge. They're cheaper and deterministic, but they need you to label the ideal chunks or IDs (reference_contexts) rather than write a reference answer, so you trade one kind of prep for another. Precision also has a no-reference LLM variant that scores chunks against the response instead.

The part that actually matters: reading them together

Any single metric is easy to misread. The skill is in the combination. Here's the diagnostic table I keep in my head. Treat these as strong hints, not laws.

Pattern Likely diagnosis Where to look High recall, low precision Retrieval finds the right info but also drags in noise Reranking, fewer chunks, better filtering High precision, low recall The chunks retrieved are clean but you're missing evidence Chunking, embeddings, retrieve more chunks Good retrieval, low faithfulness Retrieval is fine; the model is ignoring or embellishing context Prompt, generation model, temperature High faithfulness, low relevancy Grounded but not answering the question Prompt instructions, query understanding

Our policy-assistant failure reads as low recall (the paternity paragraph was never retrieved) plus a faithfulness dent (the invented "single block" rule). That points squarely at retrieval as the first thing to fix, not the prompt. Without the metrics I might have spent a day tuning the prompt, which would have done nothing, because the model never had the right document in front of it.

That's the whole value. The metrics turn "the answer is wrong" into "retrieval didn't find the evidence," which is a fixable, testable statement.

Interview angle

If you're prepping for AI or forward-deployed engineering interviews, this is high-yield. A very common question is some version of "your RAG system gives a wrong answer, how do you debug it?" The weak answer is "I'd tweak the prompt." The strong answer walks the four questions in pipeline order: did we retrieve the evidence (recall), rank it well (precision), use it faithfully (faithfulness), and answer the actual question (relevancy). Say explicitly that you check recall first, because a generation fix can't recover missing evidence. Being able to trace the data from source to answer and name what each metric isolates signals that you've actually shipped and debugged these systems.

The takeaway

Don't evaluate a RAG answer as one blob of "good" or "bad." Follow the data: did retrieval find the evidence (recall), rank it well (precision), did the model ground its answer in that evidence (faithfulness), and did it address the question (relevancy)? Two metrics about the retriever, two about the generator, in the order the data flows.

These four are the core, not the whole menu. Ragas has more (context entities recall, noise sensitivity, factual correctness, and others), and you'll reach for them as needs get specific. But if you only ever internalize these four and the difference between faithful and correct, you can debug the large majority of RAG failures.

Next article, I'll turn this into code. We'll run a real Ragas evaluation on a small policy-assistant dataset and spend most of the time interpreting the output, because the score is never the point. The diagnosis is.