Reranking: bi-encoder vs cross-encoder
Retrieval is cheap and broad. Reranking is expensive and precise. You only pay for the expensive step on a small candidate list.
| Bi-encoder (first retrieval) | Cross-encoder (reranker) | |
|---|---|---|
| Input | Query and doc embedded separately | Query and doc fed in together |
| Output | Two vectors, compared with cosine | One relevance score straight from the model |
| Speed | Fast; doc vectors are precomputed | Slow; one model call per candidate |
| Accuracy | Rough | Much better, because it sees both texts at once |
| Used on | The whole corpus (10k docs) | Only the ~100 candidates |
Common slip: a cross-encoder does not compute cosine similarity. It reads the query and the document jointly and outputs a score.
Check yourselfWhy not run the cross-encoder on all 10k docs?
It needs one model call per query-doc pair and nothing can be precomputed, so latency and cost explode. The bi-encoder narrows 10k to ~100 first.
Summary: say it in the interview
- "Bi-encoder to get recall cheaply across the whole corpus, cross-encoder to get precision on the shortlist."