What is reranking? Explain bi-encoders vs cross-encoders.

30-second answerSay your answer out loud first, then reveal.

A bi-encoder embeds the query and documents separately, so document vectors can be pre-computed. That's fast and scales to millions of documents but is less precise. A cross-encoder feeds the query and a document together through one model and outputs a relevance score. It's far more accurate but too slow to run on the whole corpus. So you retrieve a broad candidate set (say 50–100) with fast retrieval, then rerank it with a cross-encoder and keep the top 5–10.

Two-stage retrieval: a bi-encoder or BM25 pulls the top 100 candidates from the whole corpus, a cross-encoder reranker scores each query and document pair jointly, and the top 5 go into the LLM prompt.

Why cross-encoders are more accurate: attention runs across query and document tokens together, so the model can check whether the passage actually answers the question, not just whether it's on the same topic.

Why they're slow: one model pass per (query, document) pair at query time, and nothing can be pre-computed. Fine for 100 candidates, impossible for 10M.

Options: Cohere Rerank, Voyage rerank, Jina reranker, BGE-reranker (open source), and LLM-based reranking (ask an LLM to score or order passages; flexible but slower and more expensive).

Typical impact: reranking is one of the highest-ROI additions to a RAG pipeline. It often improves precision at the top of the list noticeably, which directly improves answer quality because the LLM sees better context.

Trade-offs: adds roughly 50–300ms latency and per-query cost. Mitigate by limiting the candidate count and using a small reranker.

Follow-ups to expect

What's ColBERT? A middle ground with late interaction. (See Q33.)

PreviousKeyword search (BM25) vs semantic search: why use hybrid search?

Slow is fine. Stopping is the only problem.

What is reranking? Explain bi-encoders vs cross-encoders.

Follow-ups to expect

Related