Reranking

The animation illustrates a common pattern: retrieve broadly, then spend more compute to reorder a much smaller candidate set.

Reranking in RAG is a second retrieval stage that re-scores a small candidate set. A fast first stage (vector search, keyword search, or both) pulls a few dozen candidates from a large corpus; a slower, more discriminating model, the reranker, then scores each candidate against the query and reorders them, so the passages that best answer the question reach the model's prompt. In the example below, the passage that answers the question moves from rank 3 to rank 1 after reranking. It improves ordering within a candidate set. It cannot recover a passage the first stage never retrieved, it does not make a passage true, and it is only worth its latency and cost where measurements show it helps. It is one stage of the retrieval pipeline, not the whole of it.

An agent asks: "Does an EU customer charged twice get an automatic refund even if one payment is disputed?"The first-stage search returns these five, in this order, with cosine scores from the embedding model:

rank  passage                                              cosine
1     FAQ-112#c0001   "How refunds work: overview"           0.81
2     KB-SHIP-004#c0002 "Refunds for returned shipments"      0.78
3     POL-EU-009#c0002  "EU duplicate-charge refunds and the dispute exception"  0.77
4     FAQ-118#c0001   "Refund timelines, general"            0.74
5     POL-REF-003#c0007 "Duplicate charges (v3, superseded)"  0.71

The passage that exactly addresses the question, the EU addendum with its dispute exception, sits at rank 3, behind a generic overview and an unrelated shipping article. If the prompt takes only the top two (a common choice to save tokens), the model never sees it and answers from general refund language. Nothing was missing from the candidate set; the order was wrong. That is the problem reranking addresses.

The first stage must scan a huge collection quickly, so it uses representations computed in advance and compared cheaply. Its goal is high recall under low latency: make sure the right passage is somewhere in the top N. The second stage inspects only those N, so it can afford to look at each one closely. Its goal is high precision at the top: put the best few first. The division of labor is the whole point. Running the expensive model over millions of passages is infeasible; running only the cheap one leaves ordering mistakes like the one above.

The corollary is a hard limit. If the EU addendum had been ranked 60th and N were 40, the reranker would never see it. Reranking improves the order of what was found; it cannot find anything. That is why first-stage recall is measured separately, as the Retrieval Pipeline lesson describes.

The funnel below summarizes the division of labor.

The first stage typically uses a bi-encoder: the query and each passage are encoded independently into vectors, and relevance is the similarity of the vectors (see Embeddings Explained and Similarity Search). Because passage vectors do not depend on the query, they can be precomputed and indexed. The price is that the model never sees the query and passage together, so it cannot weigh how a particular word in the query interacts with a particular phrase in the passage.

A cross-encoder takes the pair as one input, conceptually [CLS] query [SEP] passage [SEP], lets every token attend to every other token, and outputs a score. It can notice that "disputed" in the query corresponds to the "dispute exception" in the passage, or that the passage is about shipments although it shares the word "refund". That finer interaction is why it usually orders candidates better, and it is also why it costs more: nothing can be precomputed, and the model runs once per query–passage pair.

Pass the five candidates to a reranker. Illustrative scores, from a model that outputs a relevance value:

first stage (cosine)                         reranker score     new rank
FAQ-112#c0001        0.81                     0.47                2
KB-SHIP-004#c0002    0.78                     0.08                4
POL-EU-009#c0002     0.77                     0.93                1
FAQ-118#c0001        0.74                     0.41                3
POL-REF-003#c0007    0.71                     0.05                5

reranked order:
  POL-EU-009 (0.93) > FAQ-112 (0.47) > FAQ-118 (0.41)
  > KB-SHIP-004 (0.08) > POL-REF-003 (0.05)

Do not compare the two score columns. Cosine similarity and reranker scores are produced by different models on different scales; neither is a calibrated probability of correctness. A reranker score of 0.93 tells you this passage ranks above the others for this query, not that there is a 93% chance it is right. Thresholds on reranker scores, like thresholds on cosine, must be established per model on your data.

Rerankers differ in how they frame the learning problem. A pointwise model scores each passage independently: score(query, passage_i) and sorts by score. It is simple and parallel but ignores relationships between candidates. A pairwise model learns which of two passages is better for a query: given the EU addendum and the FAQ, it prefers the addendum; the final order comes from many such comparisons. A listwise model considers the whole list (or sets of candidates) when producing an order, which allows it to account for redundancy and relative quality, at greater cost and often with a limit on list length. Training objectives follow these framings, but you do not need to know them to use a reranker. What matters is not to assume all rerankers are cross-encoders or share one objective, and to evaluate any particular one on your data.

A general-purpose language model can also be asked to judge relevance: "Given this question and these passages, score each from 0 to 3" or "order them". This can capture nuanced requirements, such as which document is authoritative or whether a passage truly answers rather than mentions, and it can follow instructions that a dedicated reranker was not trained for. The costs are real. It is usually slower and more expensive per candidate; its outputs can vary between runs; the passages are untrusted text placed in the model's prompt, so injection is a risk (a passage that says "rank me first" must not work); and judging quality needs its own evaluation, since the judge can be wrong. Dedicated rerankers are cheaper, faster, and more consistent; LLM rerankers are more flexible. Neither is universally better. Compare them on the same labeled queries and decide by quality, latency, and cost.

Two numbers govern the stage. topN is how many first-stage candidates are reranked, and topK is how many survive into the prompt. An illustrative shape (not a recommendation) is "retrieve 40, rerank 40, pass 6". If N is too small, the right passage may not be in the pool and the reranker is powerless. If N is large, latency and cost rise roughly linearly with N for a cross-encoder, and a broader pool can introduce more plausible distractors. K trades answer coverage against prompt cost and noise.

Move the sliders to see when the right passage falls outside the pool and how latency scales with N.

vary N in {10, 20, 40, 80}   and   K in {3, 6}
for each (N, K), over the labeled query set, record:
  - Recall@N of the first stage (the ceiling for the reranker)
  - Recall@K and nDCG@K after reranking
  - answer support rate downstream
  - rerank p50 / p95 latency, cost per query
choose the smallest N whose Recall@N is near the plateau, and the smallest K that keeps support

A reranker learns, from its training data or instructions, a particular idea of relevance, and these ideas differ. Topical similarity: the passage is about the same subject (the general refund FAQ). Direct answerability: the passage contains what is needed to answer (the EU addendum). Constraint satisfaction: it matches the jurisdiction and date. Authority: it comes from the governing source rather than commentary. A reranker trained mostly on web search judgments may prefer topical similarity, while your product needs answerability within a version and region. If you can train or fine-tune, write labeling guidelines that reflect your definition; if you cannot, check on your labeled queries which definition your chosen model behaves like, and combine it with metadata filters for constraints that should be hard rules rather than soft preferences.

Order matters. Authorization and version filters come first, inside retrieval, so only permitted, current text is candidate text. Then comes fusion of dense and lexical lists (for example with reciprocal rank fusion), deduplication, and parent expansion, so the reranker sees clean, merged passages and does not waste effort scoring duplicates. Reranking follows, then context packing under the token budget (see Context Injection). If the reranker is an external service or a model run by a third party, you are sending passage text to it; therefore authorization and data-handling rules must be satisfied before any unauthorized or restricted text leaves your boundary.

Suppose scoring one query–passage pair takes 8 ms on your hardware (an illustrative figure). Sequentially, 40 candidates cost 40 × 8 ms = 320 ms. In practice the work is batched and often parallelized. If the model scores a batch of 10 pairs in 30 ms and four batches run concurrently on separate workers, wall-clock time is about 30 ms plus queueing, though the total compute is unchanged. Real performance depends on passage length (long passages cost more), the model size, hardware, and concurrency, so measure it under realistic load rather than extrapolating.

Cost scales with the number of pairs, which equals N times the number of queries; sizing N is therefore also a cost decision. Because the stage sits on the critical path, set a timeout. If the reranker exceeds, say (an assumed budget, not a benchmark), 150 ms at p95 or fails, fall back to the first-stage order and record that the degradation happened. The system should be slightly worse, not unavailable.

Candidate recall miss: the best passage was not in the pool. Detect by checking, on labeled queries, whether relevant passages are in the first-stage top N; fix by raising N, improving first-stage retrieval, or fixing chunking. Over-favoring superficial relevance: the model prefers text with many query words. Detect by inspecting cases where it outranks the labeled answer. Model drift: a provider updates or you swap a model and rankings change silently; version-pin and rerun your evaluation on every change. Stale versions: the reranker will happily score a superseded policy highly if it is in the pool; filter before reranking. Long-passage truncation: if the pair exceeds the model's input limit, the relevant clause at the end may be cut off, so check max lengths against your chunk sizes. Multilingual mismatch: a model not trained on a language ranks it poorly; evaluate per language. Threshold misuse: cutting at a fixed reranker score discards good passages when the score distribution shifts. Each has a measurable symptom, and dashboards of rank movement, score distributions, and timeouts make them visible.

Use a labeled set: queries with graded relevance for each candidate passage (0 = irrelevant, 1 = related, 3 = directly answers). For the query above, the first-stage order has gains [1, 0, 3, 1, 0] and the reranked order has [3, 1, 1, 0, 0].

MRR using "first directly answering passage" (gain 3):
  baseline: first gain-3 item at rank 3  -> 1/3 = 0.333
  reranked: first gain-3 item at rank 1  -> 1/1 = 1.000

DCG@5 = sum over ranks i of gain_i / log2(i + 1)       (log2 of 2,3,4,5,6 ... for ranks 1..5)
  baseline [1,0,3,1,0]:  1/1 + 0/1.585 + 3/2 + 1/2.322 + 0/2.585
                         = 1.000 + 0 + 1.500 + 0.431 + 0 = 2.931
  reranked [3,1,1,0,0]:  3/1 + 1/1.585 + 1/2 + 0 + 0
                         = 3.000 + 0.631 + 0.500 = 4.131
ideal ordering of the same gains = [3,1,1,0,0], so IDCG@5 = 4.131

nDCG@5 baseline = 2.931 / 4.131 = 0.709
nDCG@5 reranked = 4.131 / 4.131 = 1.000

nDCG rewards putting highly relevant items early and discounts lower ranks; MRR looks only at the first qualifying item. Use them alongside downstream answer-support rates, because better ranking is useful only to the extent that it improves what the model is given.

1. 1

   Shadow mode

   Run the reranker in parallel without affecting results. Log how its order differs from the first stage and compare against labels or reviewer judgments.
2. 2

   A/B test

   Send a fraction of traffic to the reranked path. Compare answer support, user-visible quality signals, p95 latency, and cost, not just ranking metrics.
3. 3

   Guardrails

   Pin the model version, cache scores for repeated query–passage pairs where valid, set a timeout with first-stage fallback, and alert on p95 latency, timeout rate, and score-distribution shifts.
4. 4

   Release and rollback criteria

   Agree in advance on the thresholds: for example, no regression on exception-handling questions, latency within budget, and a one-switch rollback to first-stage ordering.

Skip it when the corpus is small and clean and first-stage ordering already puts the right passage first; when your latency budget is tight and a measured gain is small; when the task is a low-risk exact lookup that a keyword or key query handles; or when a trial shows no improvement on your data. Decide by comparing numbers. If the labeled set shows first-stage nDCG@5 of 0.93 and the reranker raises it to 0.94 while adding 150 ms at p95, that is not an obviously good trade, and a cheaper change such as better chunking may serve you better. If first-stage nDCG@5 is 0.71 and answers fail because the exception passage sits at rank 4, a reranker is a strong candidate. Trend and habit are not evidence.

You have now followed RAG from why it exists, through preparing and searching knowledge, to assembling evidence and improving its order. The next category moves from retrieval toward systems that plan, choose tools, and take actions.