Let's say you ship a semantic search feature. Someone types "reset my password" and the top result is a doc about changing email settings. Are they related? Sure. But not what the user asked for!!
The embeddings aren't broken. The model did exactly what it was built to do — find documents that sit in roughly the same neighbourhood as the query.
This is where most retrieval systems hit a wall. If you really want to fix this, you need to take a step back and think with clarity. You will start seeing that retrieval is two jobs, not one.
Two jobs
- Find a handful of candidates out of millions — fast.
- Order those candidates so the best answer sits at the top — accurately.
A model that's great at first is usually mediocre at the second. So production systems run two different models, each built for one job: a bi-encoder and a cross-encoder.
The bi-encoder: the retriever
A bi-encoder encodes the query and each document separately into vectors.
Query → Encoder → vector
Document → Encoder → vector
Relevance is just how close the two vectors are — cosine similarity or a dot product.
The keyword is separately. Because a document never needs to see the query to be encoded, you can embed your entire corpus ahead of time and store it in a vector database. At query time, you embed one short string and run a nearest-neighbour lookup. That's why it scales to millions of documents and stays cheap.
What you give up: the query and document are compared as two finished summaries. The model never looks at them side by side. "Reset password" and "recover forgotten credentials" might land near each other, or might not, depending on how well the encoder learned the concept. Good enough to find candidates. Not always good enough to rank them.
The cross-encoder: the reranker
A cross-encoder feeds the query and document in together, as one input:
[CLS] query [SEP] document [SEP]
Now attention runs across both at once. The model can weigh "reset" against "recover," "password" against "credentials," and judge the pair directly. The output is a single relevance score.
This is far more accurate. It's also far more expensive, for a structural reason: there's no precomputing. Every query-document pair needs its own forward pass. One query against a million documents is a million forward passes — per search. Nobody does that.
Get the benefit of both
You don't pick one among them. You actually chain them.
Query → bi-encoder retrieves the top 50 from millions (fast, cheap) → cross-encoder reranks those 50 (slow, accurate — but only 50) → return the top few (5-10)
The bi-encoder cuts the problem from a million to fifty.
The cross-encoder spends its expensive attention only where it matters. You get the scale of one and the precision of the other.
What matters once you build it
The retrieval stage sets your ceiling. The reranker can only reorder what the retriever handed it. If the right answer isn't in the top 50, no amount of reranking saves you. When results feel wrong, check recall at the retrieval stage before you touch the reranker.
K is a dial. Reranking 50 candidates costs more than reranking 10. A bigger K improves the odds that the right doc is in the pile, but adds latency and compute. Tune it against real queries by running evals.
Quick Summary
It fits in one line: bi-encoders retrieve, cross-encoders rerank. One finds, the other judges.
Most good search and RAG systems run both — because the alternative is choosing between fast and accurate, and retrieval done right doesn't make you choose.
Bonus Confusion Clarity
Are Bi-encoder and Cross-encoder actually models? Are they embedding models like we discussed last time?
No. Bi-encoder and Cross-encoder are architectures/patterns, not specific models. There are several models which are trained on these architectures/patterns; you can find them in the HuggingFace leaderboard.
Bi-encoder models are usually embedding models. Example: all-MiniLM-L6-v2
Cross-encoder models are usually scoring/reranking models, not embedding models. Example: cross-encoder/ms-marco-MiniLM-L-6-v2
The two models are also trained differently — retrievers with contrastive learning, and rerankers as ranking models. That's a separate post.
Originally published on LinkedIn.