What a Cross-Encoder Re-Ranker Actually Fixes
Your RAG pipeline merges dense and BM25 results with reciprocal rank fusion (RRF) into a candidate pool of 30, then runs a cross-encoder re-ranking pass to pick the final top-5 for the prompt. A teammate asks: "RRF already produces a relevance-ordered list — why add a whole extra re-ranking pass on top of it?" What is the single most accurate answer?
A. A cross-encoder scores each (query, chunk) pair jointly in one forward pass, which is a more precise relevance signal than RRF's fusion of two independently-computed rankings. B. Cross-encoder re-ranking is computationally cheaper than RRF, so it's used to cut latency rather than to improve precision. C. Cross-encoder re-ranking retrieves from a larger candidate pool than the dense and BM25 passes did individually. D. Cross-encoder re-ranking removes the need for metadata/ACL filtering on the retrieved candidates.
Correct answer: A — "A cross-encoder scores each (query, chunk) pair jointly in one forward pass, which is a more precise relevance signal than RRF's fusion of two independently-computed rankings."
Dense retrieval scores relevance by comparing a query embedding and a chunk embedding that were computed separately — the model never sees the query and the chunk together. BM25 scores lexical overlap, also without the two texts ever being jointly encoded. RRF then fuses these two independently-computed rankings by position, which is a good, cheap way to combine two imperfect signals, but it's still built from proxies for relevance, not relevance itself. A cross-encoder takes the query and a candidate chunk together as a single input and outputs one relevance score for that specific pair — because it attends across both texts jointly, it catches distinctions (the right plan tier vs. a similar one, the right product version vs. an adjacent one) that similarity-of-separately- computed-vectors and lexical overlap both miss. That precision is also why it's applied only to a shortlist (here, 30 candidates) and not the full corpus: it's too expensive to run at index scale.
Why the distractors are wrong
- B inverts the actual cost trade-off: cross-encoder scoring is more expensive per item than RRF fusion (it's a full model forward pass per candidate), which is exactly why it only runs on a small shortlist rather than the whole retrieved set.
- C is wrong because re-ranking doesn't retrieve anything new — it re-scores the same candidate pool that dense and BM25 already produced; it narrows the list, it doesn't widen it.
- D is wrong because relevance scoring and access-control/ metadata filtering are separate concerns — filtering by tenant, locale, or ACL happens at retrieval time regardless of whether a re-ranking pass runs afterward.
Share this question