Retrieval and ranking metrics
We'll cover the following
You need a test set: queries, each with the known right chunk(s). Every metric below compares what you retrieved against that.
| Metric | Question it answers | Example |
|---|---|---|
| Recall@k | Of all relevant chunks, how many are in the top k? | 2 relevant, 1 in top 5: recall@5 = 50% |
| Precision@k | Of the k chunks I retrieved, how many are relevant? | 1 of 5 relevant: 20% |
| MRR | How high is the first relevant chunk, on average? | Rank 1 = 1, rank 2 = 0.5, rank 5 = 0.2, averaged over queries |
| NDCG | Is the whole list ordered well, with the best chunks highest? | Graded relevance, discounted by position, scaled 0 to 1 |
Details interviewers poke at
- Recall matters most. If the chunk is not retrieved, nothing downstream can save you.
- MRR is one number for the whole test set: the average of 1/rank of the first relevant chunk across queries. MRR@k is the variant that scores 0 when the first hit is beyond rank k.
- NDCG handles degrees of relevance (0 irrelevant, 1 okay, 2 perfect). A perfect chunk at rank 1 counts fully, at rank 5 much less. It is divided by the score of the ideal ordering, so it lands between 0 and 1.
- Reranker metrics are these same metrics measured after reranking. The reranker only reorders, so compare MRR and NDCG before vs after. If they did not improve, it is not earning its latency.
Check yourselfBM25 scores range from 0 to 100, right?
No. BM25 is unbounded. It depends on corpus size, query length and term rarity, so values are not comparable across queries. That is exactly why hybrid search merges ranks (RRF) and not raw scores.
Summary: say it in the interview
- "MRR only cares about the first good chunk; NDCG rewards putting the best chunks highest across the whole list."