Hybrid search and RRF
We'll cover the following
Run vector search and keyword search (BM25) together, then merge. Each one fails where the other works.
Why both
| Vector search | BM25 (keyword) | |
|---|---|---|
| Good at | Meaning. "get my money back" finds "reimbursement" | Exact rare terms. "E4021" finds the E4021 doc |
| Bad at | Codes, IDs, product names (no meaning to embed) | Synonyms and paraphrases |
The pipeline
query
├─► BM25 search ───► list 1 (top 50)
└─► vector search ─► list 2 (top 50) (bi-encoder only)
│
merge with RRF ──► ~100 unique candidates
│
cross-encoder rerank (once, at the end)
│
top 5 ──► LLMMerge first, rerank after. A reranker can only reorder what it is given. If you rerank only the vector results, the BM25-only hit never gets in.
RRF: Reciprocal Rank Fusion
BM25 and cosine scores are on different scales (BM25 is unbounded and depends on the corpus and query; cosine is roughly 0 to 1), so you cannot add them. Ranks are comparable. Each doc gets 1 / (60 + rank) from every list it appears in, and you sum.
| Doc | BM25 rank | Vector rank | Score |
|---|---|---|---|
| A | 1 | 3 | 1/61 + 1/63 = 0.0323 |
| B | 2 | not found | 1/62 = 0.0161 |
| C | not found | 1 | 1/61 = 0.0164 |
Why 60?
- The constant (k) sets how much the top ranks dominate. Small k: rank 1 overpowers everything. Huge k: rank barely matters, the merge goes flat.
- 60 comes from the original RRF paper (Cormack et al., 2009), found empirically to work well. 20 or 65 would behave almost the same.
Check yourselfA user searches "error E4021" and vector search returns junk. What fixes it?
BM25 finds the exact token. Merge it with the vector results using RRF, then rerank. Vector embeddings have no meaning for a code like E4021, so keyword search carries it.
Summary: say it in the interview
- "Hybrid for recall, reranker for precision. The 60 in RRF is a smoothing constant from the paper; I would treat it as a hyperparameter and check it against my eval set."