Debugging a wrong RAG answer
We'll cover the following
A user says "the bot gave a wrong answer". The first question is always: did retrieval fail or did generation fail? Retrieval fails more often than the LLM does, but you have to prove it.
How to tell which one broke
Look at the retrieved chunks for that exact query. That means you must log the query, the top-k chunks and the final answer for every request. Without those logs you are guessing.
| What you see | Where the problem is |
|---|---|
| The right chunk is not in the top-k | Retrieval |
| The right chunk is there, the answer is still wrong | Generation (prompt or model) |
Why the right chunk goes missing
- Bad chunking: the answer is split across two chunks, or chunks are too big (diluted) or too small (no context).
- Wording mismatch: the user says "refund", the doc says "reimbursement". Pure keyword search misses it. Conversely, vectors are weak on exact tokens like error codes.
- Top-k too small, no reranker: the right chunk is at rank 12 and you only pass the top 5.
- Stale or wrong index: the document was updated and the embeddings were not.
Check yourselfThe right chunk is in the top 5 but the answer is wrong. Retrieval or generation?
Generation. Retrieval did its job, so look at the prompt (is it told to use only the context?), the context order, and the model.
Summary: say it in the interview
- "First I check whether the right chunk was retrieved. If it was not, it is a retrieval problem; if it was, it is generation. That is why I log query, chunks and answer for every request."