Generation metrics (Ragas)
We'll cover the following
Three families: retrieval metrics, ranking metrics, and generation metrics. Generation ones are mostly LLM-as-judge.
| Metric | What it checks |
|---|---|
| Faithfulness | Is every claim in the answer backed by the retrieved context? Catches hallucination. |
| Answer relevancy | Does the answer address the question that was asked? |
| Answer correctness | Does it match a reference answer? Needs ground truth. |
| Context precision / recall | Ragas scores retrieval too, using an LLM instead of labels. |
| Completeness | Does it cover every part of the question? |
| Helpfulness, coherence, tone | Subjective; custom rubrics judged by an LLM. |
| Safety | Toxicity, PII leakage. |
Read the two together
| Recall@k | Faithfulness | Fix |
|---|---|---|
| Low | any | Retrieval: chunking, hybrid search, reranker |
| High | Low | Generation: the prompt, context order, or model is ignoring or adding to the context |
Check yourselfRecall@5 is 90% but faithfulness is 60%. Where is the problem?
Generation. The right context is being retrieved, but the model adds claims that the context does not support. Tighten the prompt, shorten and order the context, or change the model.
Summary: say it in the interview
- "Low recall means fix retrieval. High recall with low faithfulness means the problem is the prompt or the model."