Retrieval Pipeline

The animation walks through one example. This page owns the broader production pipeline: candidate generation, filters, ranking, deduplication, context selection, and fallback behavior.

Hybrid search in RAG runs keyword search (BM25) and vector search side by side and merges their results. Reciprocal rank fusion (RRF) does the merge using only ranks: each document scores the sum of 1 / (k + rank) over every list it appears in, so a chunk found by both methods outranks a chunk found by only one, without comparing BM25 and cosine scores directly. This is the heart of the RAG retrieval pipeline, the online half of the system described in the RAG Pipeline lesson.

A production retrieval pipeline turns a user's question into a short, ordered, authorized set of evidence. It understands the question, applies identity-based filters, gathers candidates through several methods (dense vectors, keyword search, exact lookups), fuses their rankings, removes duplicates, reranks, and selects what fits in the token budget. Each stage can be wrong in its own way, so the pipeline is designed to be observable: every stage's input and output can be logged and replayed, and optional stages are added only when measured errors justify them.

Consider: "Which refund policy applies to an EU customer charged twice on order 8472 after the October policy update?" A naive pipeline embeds the whole sentence and returns the top five chunks. The result is likely a mix of general refund overviews, because the embedding captures the broad topic. What it lacks is everything that makes this question specific. Jurisdiction (EU) selects a particular addendum. Time (after the October update) selects a policy version by effective date. The order identifier 8472 is not policy text at all; it is a key into a live transaction system. And the word "applies" signals that the user wants a determination, which depends on facts the documents cannot supply.

A retrieval pipeline exists to carry those constraints through search instead of hoping that a single embedding preserves them. This lesson follows this question through the stages, and the final trace shows each intermediate result.

Before building stages, define the interface. Inputs: the user's question; the authenticated identity (user ID, tenant, groups); locale; the current time; any explicit filters from the UI; and a retrieval budget (maximum candidates, tokens, and milliseconds). Outputs: an ordered list of evidence candidates, each with chunk ID, source ID, version, offsets, text, scores from each stage, and the reason it was selected. Everything in the output has already passed authorization. Stating this contract matters because it lets you test the pipeline without the language model and swap stages independently.

The contract also draws a boundary between two kinds of lookup. Document search returns passages from the knowledge base. Transaction lookup fetches a specific record, such as the charges on order 8472, from an authoritative system through an authorized API. The first is probabilistic; the second is deterministic and must be executed by trusted code with the user's permissions. The pipeline can run both, but it must never answer the second by searching the first.

The first stage converts free text into structured constraints where it can, and leaves the rest as text for search. For our question a deterministic parser (regular expressions, a small entity dictionary, and the request's locale) can find order_id = 8472, the region keyword EU, and the phrase "October policy update", which maps to a date range beginning 2026-10-01. Deterministic extraction is cheap, fast, testable, and never invents anything. It fails on phrasing it was not written for.

An LLM can do the extraction more flexibly, handling "the update last month" or "customers in Germany", and can also rewrite the query. The price is added latency (often hundreds of milliseconds for a model call), cost, nondeterminism, and risk: the model may drop or alter a constraint. A rewrite that turns "after the October update" into "recent updates" has silently weakened the search. Whenever an LLM produces constraints, validate them against what the original text contains (for example, confirm that the order ID in the output appears in the input) and keep the original query as one of the searches.

Rewriting makes a query a better search input; decomposition splits a multi-part question into subqueries that can be answered separately. For our question:

original:   "Which refund policy applies to an EU customer charged twice on order 8472
             after the October policy update?"

constraints: region=EU, order_id=8472 (-> tool), effective_after=2026-10-01

subquery 1: "refund policy for duplicate charges EU consumers"      (policy text)
subquery 2: "refund policy changes October 2026 effective date"      (version/changelog)
tool call:  get_order_payments(order_id=8472, as=user)               (live facts, not search)

The results of the subqueries must be linked back: the final evidence should note which subquery each passage answers, so the prompt can keep the two roles distinct. Decomposition multiplies retrieval calls and so latency and cost, and each subquery can fail independently. Rewriting can also improve a poorly phrased query or worsen a good one. Treat both as options to be turned on only where evaluation shows a win, and always compare against retrieval with the unmodified query.

Filters must be part of the retrieval, not an afterthought. The filter is constructed from trusted inputs, the authenticated identity and the system's notion of version validity, and it is passed to every candidate generator.

{
  "tenant_id": "acme-eu",
  "acl_any_of": ["support-agent"],            // groups of the authenticated user
  "status": "current",
  "effective_date": { "lte": "2026-10-28" },  // not a future-dated draft
  "region": ["EU", "ALL"]                     // from the extracted constraint
}

There are two ways to apply permissions, and one of them is a trap. Pre-filtering restricts the search to permitted records, so every returned item is authorized and you get k results if enough exist. Post-filtering retrieves the top k from everything and then drops unauthorized items. It can leave you with too few results, and it also means unauthorized text was handled by the search layer and, if done carelessly, by reranking or prompt code. In particular, never send text a user may not read to an external reranker or model, only to remove it later. Instructions in the prompt such as "ignore documents the user lacks access to" are not enforcement; by that point the text has been disclosed to the model.

The figure below contrasts the two on a top-5 search (illustrative).

Dense vector retrieval finds passages whose meaning is close to the query, including those with different wording. Lexical retrieval (BM25 or similar) scores literal term overlap, weighting rare terms more heavily; it excels at exact phrases, codes, and names. Exact lookups and metadata filters fetch by identifier or attribute with certainty. For our question, the three return different things:

dense (cosine, top 4)                       lexical (BM25, top 4)
1  POL-EU-009#c0002   EU duplicate addendum   1  POL-REF-004#c0007  "duplicate charges" v4
2  FAQ-112#c0001      refund overview         2  RB-PAY-017#c0003   "duplicate capture" runbook
3  POL-REF-004#c0007  duplicate charges v4    3  POL-REF-004#c0011  changelog: "October 2026"
4  RB-PAY-017#c0003   duplicate capture       4  POL-EU-009#c0002   EU addendum

exact / tool:  orders API -> order 8472: payments=[capture#1 settled, capture#2 settled]  (live)

You now have two ranked lists. Cosine scores and BM25 scores live on different scales (cosine in [−1, 1], BM25 unbounded and dependent on corpus statistics), so adding or averaging them is meaningless. Reciprocal Rank Fusion (RRF) sidesteps this by using only ranks: score(d) = Σ 1 / (k + rank_i(d)), summed over the lists in which d appears (Cormack, Clarke & Büttcher, SIGIR 2009; BM25 is the Robertson–Spärck Jones probabilistic ranking function). The vector side is covered in Similarity Search. The constant k dampens the advantage of top ranks; 60 is a commonly used example value and is configurable, not a rule.

POL-REF-004#c0007:  dense rank 3, lexical rank 1  -> 1/63 + 1/61 = 0.015873 + 0.016393 = 0.032266
POL-EU-009#c0002:   dense rank 1, lexical rank 4  -> 1/61 + 1/64 = 0.016393 + 0.015625 = 0.032018
RB-PAY-017#c0003:   dense rank 4, lexical rank 2  -> 1/64 + 1/62 = 0.015625 + 0.016129 = 0.031754
FAQ-112#c0001:      dense rank 2 only             -> 1/62                               = 0.016129
POL-REF-004#c0011:  lexical rank 3 only           -> 1/63                               = 0.015873

fused order: POL-REF-004#c0007 > POL-EU-009#c0002 > RB-PAY-017#c0003 > FAQ-112#c0001 > POL-REF-004#c0011

Agreement between independent retrievers is evidence of relevance, and RRF rewards it without needing score calibration. It does not know which retriever was right, and it does not know about authority or recency. That is what the next stages add. Weighted variants exist; tune them only against labeled results.

Change k to recompute the fused scores from the two lists above.

Fused lists commonly contain the same content more than once: overlapping chunks from the same document, the same passage reached through two subqueries, or the same text mirrored in a wiki and a PDF. Duplicates waste prompt tokens and skew the model toward repeated claims. Collapse chunks with the same source and version whose offsets overlap or touch into a single span; drop exact or near-duplicate text keeping the highest-authority copy; and, where the index stores small child chunks, expand each retained hit to its parent section so that rules and their exceptions travel together (see Chunking Strategies). Every merged or expanded item must keep its source, version, offsets, and ACL, and expansion must re-check authorization on the parent.

Fusion gives a decent candidate order by cheap signals. A reranker, covered in depth in the Reranking lesson, reads the query together with each remaining candidate and reorders them by a finer judgment. At the interface level, it takes a query and a list of authorized passages and returns the passages with a new score or order; it cannot add passages that were never retrieved.

Selection then fits the reranked list into the evidence budget. The aim is coverage, not just the top scores. If a passage states the rule and another states an exception or an older conflicting version, both should survive even if one scores lower, because dropping a qualifier produces a confident wrong answer. How the selected evidence is formatted and ordered for the model is the subject of Context Injection.

1 parsed       region=EU  order_id=8472  effective_after=2026-10-01  intent=policy_applicability
2 queries      Q1 "refund policy for duplicate charges EU consumers"
               Q2 "refund policy changes October 2026 effective date"
               TOOL get_order_payments(8472)
3 filter       tenant=acme-eu  acl_any=[support-agent]  status=current  region in [EU, ALL]
               -> 1,284 eligible chunks (removed: 3,912 other-tenant/other-ACL/superseded)
4 dense        POL-EU-009#c0002, FAQ-112#c0001, POL-REF-004#c0007, RB-PAY-017#c0003
  lexical      POL-REF-004#c0007, RB-PAY-017#c0003, POL-REF-004#c0011, POL-EU-009#c0002
5 authorized   all 5 unique chunks pass the ACL check again at fetch time
6 fused (RRF)  POL-REF-004#c0007 > POL-EU-009#c0002 > RB-PAY-017#c0003 > FAQ-112#c0001 > POL-REF-004#c0011
7 dedupe       POL-REF-004#c0011 (changelog) is adjacent to c0007; merged span 5120-6340
8 reranked     POL-EU-009#c0002 (0.91) > POL-REF-004#c0007+c0011 (0.88) > RB-PAY-017#c0003 (0.62) > FAQ-112#c0001 (0.20)
9 selected     3 passages, 1,640 tokens (budget 4,000); FAQ-112 dropped as general
tool result    order 8472: two settled captures (live, authoritative)

Notice what the trace gives you. If the answer later cites the wrong policy, you can see whether the right passage was in the eligible set (stage 3), found by either search (4), lost in fusion or dedupe (6, 7), or pushed out by the reranker or the budget (8, 9). Without the trace, every failure looks the same.

If the filtered search returns nothing relevant, a fallback to lexical-only search can help when the vector side failed, but widening the filter (dropping the region constraint or the ACL) is not an acceptable fallback. When the question is underspecified ("What's the policy?"), asking a clarifying question is often better than guessing. If the evidence retrieved is from a document version that is not current, or the index timestamp for the document is older than the source's modification time, treat the result as potentially stale: either re-fetch, answer with a stated caveat, or decline. If two current passages conflict (a regional addendum and a global policy), pass both with their scope and dates, and let the instructions say which takes precedence.

Retrieval sits on the critical path to the first token. A latency budget makes the arithmetic explicit. The values below are illustrative.

query understanding (deterministic)          15
dense search                                 40  \
lexical search                               25   > run concurrently: max(40, 25, 60) = 60
order lookup via tool                        60  /
fusion + dedupe                              10
rerank 30 candidates                        120
pack evidence                                10
-------------------------------------------------
critical path                    15 + 60 + 10 + 120 + 10 = 215

if the three searches ran one after another: 15 + (40+25+60) + 10 + 120 + 10 = 280
adding an LLM rewrite step (e.g. 250 ms) before retrieval would make it 465

Beyond arithmetic, plan for timeouts per stage with sensible defaults (skip reranking, use first-stage order); caches for repeated queries and for query embeddings, invalidated when documents or permissions change; concurrency limits and rate limits on the embedding and reranking services; and fan-out control for decomposed queries so that one user question cannot trigger dozens of searches. Degrade deliberately: losing the reranker should lower quality slightly, not take the system down, while losing the authorization filter must fail closed.

Build an offline set of questions with the passages (and tool results) required for each. Track Recall@K at the candidate stage, and rank-sensitive measures such as MRR (or nDCG where relevance is graded) after reranking, where they are justified by the question mix. Add operational metrics: the authorized-hit rate (the fraction of retrievals that return at least one permitted result), the zero-result rate, freshness lag, p95 latency per stage, and the quality of rewrites (does the rewritten query preserve the constraints?). Online, collect signals such as whether users open citations, rephrase, or give negative feedback, but treat them as noisy. Always log a trace ID connecting question, stage outputs, selected evidence, and answer, taking care that logs themselves respect access rules.

Begin with the simplest pipeline that has authorization: a filtered search. If users mostly search with distinctive keywords and identifiers, a lexical query may be all you need. Add dense retrieval when error analysis shows relevant passages missed because of vocabulary differences, and fuse them when each finds things the other misses. Add a reranker when the right passage is usually in the candidate set but often not near the top. Add decomposition when questions routinely span several topics. Add rewriting when queries are underspecified or conversational. In each case, justify the stage by a counted failure in your evaluation data, and keep the ability to switch it off.

The pipeline produces evidence. The next lesson turns that evidence into a prompt that is useful, budgeted, and resistant to manipulation.