Enterprise RAG Knowledge Assistant System Design: Complete Architecture

An enterprise RAG (Retrieval-Augmented Generation) knowledge assistant answers employees' questions from their company's own documents — wiki pages, shared drives, chat — with cited answers, while showing each person only what they are allowed to read. The final design has two paths. An asynchronous ingestion path pulls content and permissions from each source through connectors and a durable queue, parses and chunks every document, embeds the chunks, and writes them with version fencing into tenant-partitioned lexical (BM25) and vector (HNSW) indexes, tagging every chunk with its revision status and the principals allowed to read it. A synchronous query path resolves who is asking, runs hybrid retrieval with the permission filter pushed into both indexes, reranks the candidates, re-checks access live, and lets an LLM answer only from the surviving evidence — validating every citation, or abstaining when the evidence is insufficient. This page builds that design one stage at a time, starting from keyword search plus an LLM.

The problem

Employees ask questions whose answers live in internal documents: "What changed in our payment reconciliation process last month?" or "Which runbook covers a rejected settlement file?". The answers are spread across thousands of pages in several systems, they change every week, and each page has its own audience. A useful assistant has to find the right passage, use the current version of it, respect who is asking, and show where the answer came from.

Why a plain LLM fails

A general-purpose language model was never trained on a company's private documents, and its training data stops at a cutoff that predates "last month" by definition. Asked the question above, it either says it does not know — honest but useless — or produces a fluent, confident answer that is invented. Fluency and correctness are separate properties of model output.

Why fine-tuning is not the fix

Training or fine-tuning a model on the corpus looks like the next idea, and it fails on four counts that matter here. Freshness: a training run takes hours to days and would have to repeat for every edit, while documents change daily. Provenance: knowledge blended into weights cannot be cited, so nobody can check where a claim came from. Authorization: one set of weights serves every user, so anything learned from a restricted document is available to everyone who asks — weights cannot be filtered per user. Deletion: a document removed at the source cannot practically be removed from a trained model. Fine-tuning is still useful for tone, output format or domain vocabulary; it is the wrong place to store knowledge.

Pasting the whole corpus into a long-context model is the other shortcut. At 150,000 documents of ~1,500 tokens each, the corpus is ~225 million tokens — orders of magnitude beyond any context window, and paying for it on every question would be absurd even if it fit. Trade-offs → Long-Context Stuffing vs. Retrieval covers the small-corpus cases where long context does win.

What RAG does instead

Retrieval-Augmented Generation leaves the model's weights alone. For each question, the system retrieves a few relevant passages from an index of the company's documents — only passages the asker may read, and only from current revisions — and places them in the model's prompt with an instruction to answer only from that evidence and cite it. The model's job changes from recalling facts to reading supplied evidence. That job is checkable, it updates as soon as the index updates, and it can be restricted per user before the model sees anything.

The running example

One small example makes the mechanisms concrete throughout. Tenant acme connects its wiki, shared drive and team chat to the assistant. Its payment-reconciliation policy, document POL-17, exists in two revisions: r2 says reconciliation runs manually at month end; r3, published last month, says it runs automatically every night with an exception queue for unmatched transactions. A runbook, RUN-8, describes the operational steps. The policy and runbook are readable by the group grp-payments-eng; a user who is only in grp-sales must not see them. The example question is "What changed in our payment reconciliation process last month?". A second tenant, globex, appears only where tenant isolation is discussed.

Four ways a RAG answer can be wrong

A credible enterprise RAG system has to get four largely independent things right, and a failure in any of them looks the same to the user — a wrong or untrustworthy answer — while needing a completely different fix. Retrieval relevance: did it find the passages that answer the question? Authorization: did it use only passages this user may read right now? Freshness: did it use the current revision, not a superseded or deleted one? Groundedness: does the generated text say only what the evidence says, with checkable citations? Each stage in this design closes one or more of these gaps, and the guarantees chapter (Chapter 17) returns to why fixing one never fixes another.

The retrieval-augmented generation architecture on this page uses a small vocabulary, assembled in the final architecture. Each term is defined once here and reused without redefinition.

Functional requirements describe what the assistant must do. The essential MVP is separated from clearly labelled extensions so the design stays focused on what makes an enterprise assistant trustworthy rather than a demo.

Connect sources and keep them in sync

A tenant administrator connects a source system — wiki, shared drive, team chat — and the system pulls the initial corpus and every later change: new documents, new revisions (POL-17 r2 → r3), deletions, and permission changes. Each connector must fetch the source's own access-control information together with the content; content without its permission context cannot be ingested safely.

Index every revision with its permissions

Every ingested revision is parsed, chunked, embedded and written to a lexical and a vector index, scoped to its tenant, with its ACL and revision status attached at ingestion time — never added after content is already searchable.

Answer with citations, or abstain

An authenticated user asks a natural-language question and receives a streamed answer built only from evidence they may currently read, citing the chunks it relied on. For the running question, a payments engineer gets an answer grounded in POL-17 r3, noting that r2 is superseded and citing RUN-8 for the operational steps. When the readable evidence does not answer the question, the system says so instead of guessing.

Enforce authorization on every answer

No answer, citation or paraphrase may reveal content the requester cannot currently read, even though that content is in the index. A user in grp-sales asking the same question receives "I don't have access to information that would answer that." — never a summary of the policy, and never a hint that a restricted document exists.

Feedback, lifecycle, and audit

Users can rate an answer, tied to the request that produced it. Administrators can tombstone documents and see sync status. Every answer leaves an auditable trace of what was retrieved, what passed authorization and what was cited.

For a RAG system the non-functional requirements that matter most are not raw throughput. They are authorization, tenant isolation and freshness, stated precisely enough to test, with latency and cost as budgets inside those constraints.

Authorization: a hard invariant, fail-closed

Authorization correctness is not a percentile. No answer, citation or paraphrase may reveal content the requester cannot currently read — under any load and with any dependency degraded. When an authorization check cannot be completed confidently (timeout, missing record, unclear membership), the content is treated as unreadable: the system fails closed. A false deny costs a retry; a false allow is a leak that cannot be undone.

Revocation: a stated propagation budget

"Immediately" is not achievable across distributed systems, so revocation gets a number. As illustrative targets for this walkthrough: a removal from a group takes effect within the identity-cache TTL (≤ 60 s), because group membership is resolved per query; a document-level permission removal takes effect within 2 minutes p95 of the source emitting it, via a priority event lane. Every cache that could serve restricted content must have a TTL inside this budget.

Tenant isolation: structural, not best effort

One tenant's queries must never retrieve, rank or even touch another tenant's documents. That is enforced by partitioning storage and indexes per tenant, not only by a query-time WHERE clause that one code path could forget (Chapter 11).

Freshness: index lag as an SLO

Index lag is the time from a source edit to that edit being retrievable. Illustrative targets: p95 within 5 minutes for small edits and chat messages, within 15 minutes for large documents that need full re-chunking and re-embedding, with a nightly reconciliation sweep as a backstop for missed events. A system that cannot state this number has no freshness contract.

Latency, availability, groundedness, durability

Time to first token should feel conversational: p95 under 1 s, of which everything before generation (retrieval, reranking, authorization) takes ~400 ms or less; the full answer streams over a few seconds. The query path targets 99.9% availability; ingestion may pause without violating it. At least 95% of factual claims should be traceable to a cited, readable chunk in sampled evaluation — less than 100% because validation reduces, but cannot eliminate, unsupported sentences. Ingestion is durable (an accepted revision is never silently lost) and idempotent (a re-delivered event has no second effect).

Every number here is an illustrative design assumption, not a benchmark. The point is the estimation method and the architectural conclusions it forces — above all, that index size is small and LLM tokens dominate cost.

Corpus and chunks

Assume the largest tenant has 150,000 documents across wiki, drive and archived chat, averaging 1,500 tokens each. With 400-token chunks and a 100-token overlap (a stride of 300 tokens between chunk starts), each document yields ceil(1,500 ÷ 300) = 5 chunks, so the tenant has 150,000 × 5 = 750,000 chunks.

Index storage

At 1,536 dimensions stored as 32-bit floats, one vector is 1,536 × 4 = 6,144 bytes (~6 KB), so raw vectors take 750,000 × 6.1 KB ≈ 4.6 GB. An HNSW graph adds its neighbour links on top — with M = 16-32 that is roughly 100-200 MB, only a few percent at this dimensionality — for a vector-index footprint of ~5 GB (plan ~6-7 GB with merge overhead and headroom, per replica). Chunk text plus metadata (ids, offsets, revision status, ACL tags) at ~2.5 KB per chunk (a 400-token chunk is ~1.6-2 KB of text) adds 750,000 × 2.5 KB ≈ 1.9 GB.

Ingestion churn

Assume 1,200 revisions/day for the tenant (policy updates, runbook edits, newly archived chat threads). Each revision re-chunks and re-embeds only its own document, so steady-state load is 1,200 × 5 = 6,000 chunk embeddings/day — a trickle. The real ingestion spikes are the initial sync (750,000 chunks) and an embedding-model migration, which re-embeds the whole corpus (Chapter 12).

Query volume

Assume 3,000 employees, 1,100 daily active users, 4 questions/day each: 4,400 queries/day ≈ 0.05 queries/s on average. With an 8× business-hours peak, the tenant peaks at ~0.4 queries/s. The platform serves an illustrative 500 tenants of very different sizes, so a platform-wide peak of 150-200 queries/s is the planning figure, dominated by the largest tenants active at the time.

LLM tokens per query

Per query: a rewrite step on follow-up turns (~50 input / ~30 output tokens), 4 reranked chunks × 400 tokens ≈ 1,600 tokens of context, ~500 tokens of system prompt and conversation history, and the question — about 2,200 input and 350 output tokens, ~2,550 in total. For the tenant: 4,400 × 2,550 ≈ 11.2 million tokens/day; across 500 tenants, an order of magnitude in the billions per day. Context tokens — chunk size × chunk count — are multiplied by every query, so they are the largest controllable lever on cost and latency.

Latency budget

What the numbers decide

The index is small. ~5 GB of vectors fits in memory on one node, so the largest tenant's index does not need sharding for size; partitions exist for isolation and blast radius, and replicas for throughput and availability. The LLM tier is the throughput bottleneck. At 150-200 queries/s and ~5 s of streaming per answer, roughly 1,000 generations are in flight at peak — provider quotas or GPU capacity, not retrieval, set the ceiling. Ingestion is light except for bulk events, so it can run on a separate, queue-driven pool. Context size is the cost lever, which is why Chapter 10 spends a reranker to keep the prompt to four chunks.

Start with the smallest thing that could work: copy one source's pages into a table, match keywords with SQL and paste the matching documents into the prompt. It is a fair prototype — and each way it breaks names a stage of the final design.

The design

A nightly job exports every wiki page into one table. A question is reduced to keywords, matched with LIKE, and the first few matching pages are concatenated, in full, into the LLM prompt.

When it works

For one small team, one source and documents everyone may read, this is a reasonable first week of work: it answers some questions, it is easy to reason about, and it shows whether people want an assistant at all.

Where it breaks

The roadmap these limits create

Each remaining chapter is one architectural change, made because the previous stage hits a specific limit:

The baseline's data problems — a stale copy, whole documents, no revision identity, no permission data — all come from how content gets in. Stage 1 replaces the nightly export with a pipeline that keeps a current, chunked, versioned, permission-tagged copy of every source. Search is still keyword-based, but over chunks and with a real ranking function.

Connectors: content and permissions in the same pull

One connector per source per tenant authenticates with the source's API and turns its changes into events: revision upserted, document deleted, permissions changed. Webhooks give low latency; a cursor-based poll catches what webhooks miss; a nightly reconciliation sweep compares the source listing with the index as a backstop (Trade-offs → Sync Trigger). The connector fetches the document's ACL in the same pull as its content: permissions cannot be reconstructed later, and content without them cannot be served safely.

A durable queue between connectors and processing

Connectors are bursty — an initial sync emits 150,000 documents at once — while parsing and embedding are slower and rate-limited. A durable queue absorbs bursts, retries failures and lets each side scale independently. A backed-up queue degrades freshness (index lag rises and alerts), never query latency. Delivery is at-least-once, so every consumer must tolerate duplicates and out-of-order events.

Parse, normalise, chunk

The parser extracts clean text from each source format while keeping structure (headings, lists, tables), and computes a content hash so a metadata-only edit with identical text is skipped. The chunker splits the text into ~400-token chunks, preferring heading boundaries and overlapping neighbours by ~100 tokens. Each chunk gets an id scoped to its revision (POL-17-r3#c2); revisions never share chunk rows. Chunk size and overlap become important once chunks are embedded, so Chapter 08 revisits them.

Idempotent, version-fenced writes

Because events can repeat or arrive late, the index writer applies two checks before changing anything. Idempotency: an event for a (doc_id, revision_id) that is already applied is acknowledged and dropped. Version fencing: a revision is applied only if its source version is newer than the one recorded for the document. If a backfill job replays the r2 publish event after r3 is live, the guarded update matches zero rows, nothing is written, and r2 cannot resurrect.

The sync cursor advances only after a revision's chunks, index entries and document pointer are all written. Advancing it after a partial write would leave the indexes silently inconsistent with no event left to repair them. When a second index arrives in Stage 2, the same rule covers both.

The data model

The model separates what the source says exists (document, revision, ACL), what the assistant has indexed (chunks with tags), and what an answer was built from (trace, citations).

Two constraints carry the correctness: (tenant_id, doc_id, source_version) is unique, which makes replays detectable; and chunk ids are scoped to a revision, so r2's and r3's chunks are different rows and history survives.

What changes in the request flow

Ingestion is now continuous and asynchronous. The query path searches a BM25 index of chunks instead of scanning whole documents, so results are ranked, the prompt receives a few focused passages, and every passage has a stable id that later stages can cite.

What Stage 1 does not fix

BM25 still matches words, not meaning, so vocabulary mismatch remains. Permission data and revision status are stored but not yet enforced at query time. The next stage tackles relevance.

BM25 ranks passages, but only by shared words. Stage 2 adds an embedding model and a vector index so retrieval can match meaning, and it is the stage where chunk size and overlap start to matter.

The limit it addresses

"How are bank payments matched now?" and the chunk "Reconciliation now runs automatically every night; unmatched transactions go to an exception queue" share almost no terms, so BM25 scores the chunk near zero. Their embeddings are close, because they are about the same thing.

How it works

At ingestion, the embedder converts each chunk into a 1,536-dimension vector, in batches for throughput, and the index writer stores it with the chunk id and tags. At query time, the question is embedded with the same model (~30 ms) and the vector index returns the ~30 nearest chunks by cosine similarity. An exact scan would read all ~4.6 GB of vectors for every query; HNSW walks a neighbour graph and visits a few thousand vectors instead, trading a small, tunable recall loss for millisecond latency (Architecture Deep Dive → Retrieval Internals).

Chunk size and overlap now matter

A chunk becomes one vector, so its size decides what that vector means. Too large, and one vector averages several topics, matching everything weakly and spending context tokens on irrelevant text. Too small, and a chunk loses the context that makes it meaningful ("it now runs nightly" — what does?). Structure-aware chunking along headings keeps chunks coherent at the cost of uneven sizes; fixed-size windows are predictable but cut sentences; semantic chunking splits where the topic shifts, at higher ingestion cost (Trade-offs → Chunking Strategy).

Overlap repeats the end of one chunk at the start of the next, so a sentence that straddles a boundary survives intact in at least one chunk. It is not free: with a 100-token overlap the 1,500-token document needs 5 chunks instead of 4, ~25% more vectors and storage, and adjacent chunks can both reach the shortlist with mostly the same text, which context assembly must de-duplicate (Chapter 10).

The embedding model is part of the data

Vectors from different models live in different spaces and cannot be compared, so every chunk records which model produced it, and queries must be embedded with the model that built the index they search. Changing models therefore means re-embedding the whole corpus — a lifecycle operation covered in Chapter 12.

What changes in the request flow

The question is now embedded before search, and the vector index replaces BM25 as the retriever. Replacing keyword search outright is the tempting move — and the next section shows why it is a mistake.

Where vectors fall short

Embeddings capture topic, not exact strings. RUN-8 and RUN-5 are both short runbook identifiers about settlements, so their chunks embed almost identically, and "What does RUN-8 say about rejected files?" can rank the wrong runbook first. The same applies to error codes, ticket numbers, SKUs and rare names — precisely the tokens employees paste into questions. Vectors also know nothing about revisions or permissions.

Vector search finds paraphrases; BM25 finds exact tokens. Each misses what the other catches, so Stage 3 runs both on every query and merges their rankings.

How hybrid retrieval works

The retriever sends the question text to the lexical index and the question embedding to the vector index in parallel, ~30 results each, with identical filters. The two lists are merged with reciprocal rank fusion (RRF), which scores each chunk by its rank in each list rather than by raw scores. BM25 scores and cosine similarities live on incomparable scales; ranks do not, so RRF needs no normalisation, and a chunk ranked well by either method rises. The result is ~40 unique candidates. For "What does RUN-8 say about rejected files?", BM25 puts RUN-8 chunks first on the exact token while vector search contributes passages that paraphrase the question.

Alternatives

Weighted score blending (α·BM25 + (1-α)·cosine) needs per-corpus score normalisation and drifts when either scorer changes. Vector-only with keyword boosting patches identifiers case by case. Learned sparse retrieval models such as SPLADE combine both signals in one index at higher ingestion cost. RRF wins here as the simple, robust default (Trade-offs → Retrieval Method; Architecture Deep Dive → Hybrid Fusion and Reranking).

One engine or two?

The two indexes can live in one search engine that stores the text, the vector and the tags on the same record, so a chunk is written once; or in two stores, which then need a shared commit point (the fenced writer of Stage 1) and a reconciliation job that compares them. Two stores updated independently drift: a chunk deleted from one but still returned by the other is a classic, hard-to-detect bug.

What remains

Fusion produces a good candidate pool, not a good prompt. Forty candidates × 400 tokens is ~16,000 tokens — ten times the context budget — and the first-pass scores are too coarse to choose the best four reliably. That is Stage 4.

Stage 4 adds a cross-encoder reranker and a fixed context budget, so the model sees a few strong passages instead of forty plausible ones.

Why first-pass scores are not precise enough

Both first-pass methods score the question and each chunk independently: vector search compares two embeddings computed separately, BM25 counts terms. Neither can notice that the question asks what changed last month, so a glossary entry that merely mentions reconciliation can rank next to the passage that describes the change. A cross-encoder reads the question and a chunk together and scores how well that chunk answers that question. It is far more precise and far more expensive — one model inference per pair — so it runs only on the ~40 fused candidates (~120 ms), never on the corpus.

The context budget

The capacity estimate allows 4 chunks × 400 tokens ≈ 1,600 context tokens per query. Each additional chunk adds ~400 input tokens to every query (~18% more input cost at ~2,200 input tokens) and more latency. More context is also not monotonically better: marginal passages distract the model, and facts in the middle of a long context are used less reliably than those at the edges. The budget is tuned with context precision and recall@k (Chapter 14), not maximised.

Assembling the context

The context builder takes the reranked list in order, drops near-duplicates created by chunk overlap, caps chunks per document so one long page cannot crowd out everything else, and labels each chunk with a citation marker ([1], [2]...) mapped to its chunk id.

Alternatives

Fusion only (no reranker) saves ~120 ms and a model, at a measurable precision cost. Conditional reranking skips the reranker when one candidate clearly dominates. LLM-as-reranker is more flexible but slower and costlier per pair (Trade-offs → Reranking).

What remains

Retrieval now finds the right evidence — for everyone. The pipeline returns the payments policy to a sales user exactly as readily as to a payments engineer, and nothing stops one tenant's search from touching another's data. Stage 5 fixes that, and it changes the retrieval stages built so far.

Authorization is not a filter added once retrieval works. It decides how data is partitioned, what every indexed chunk carries, how the retriever queries the indexes, and what happens just before generation. Stage 5 reworks the earlier stages under that constraint.

Two boundaries

There are two different boundaries. Tenants must never see each other's data, and that boundary never changes per user. Document permissions inside a tenant differ per user and change over time. They need different mechanisms.

Tenant isolation: a partition, not a WHERE clause

If isolation were a tenant_id = :tenant predicate every query must remember, one missed predicate in a new feature, an admin tool or a debugging script would leak another tenant's documents. Instead, each tenant's lexical and vector data live in their own partition (a separate index or namespace), and the authenticated tenant of the request selects the partition. A query for acme cannot return globex data because it never searches globex's partition. Partitioning bounds data exposure; it does not by itself stop a noisy neighbour from consuming shared compute — that needs per-tenant quotas or reserved capacity (Chapter 14).

The simplest ACL approach: post-filtering, and why it fails

The obvious design retrieves the top 30 as before and then drops chunks the user cannot read. It fails in three ways. Starvation: for a sales user, most of the top 30 for a payments question are restricted; two or zero survive, even when a readable answer exists at rank 200, and raising k only moves the cliff. Fragility: every code path that reads the index must remember the filter. Wasted work: the rerank budget is spent on chunks that will be thrown away.

The next idea, filtering by doc_id IN (documents this user can read), pushes the check into the search but needs a per-user list that can hold tens of thousands of ids, rebuilt whenever any permission changes.

Pushdown: principal tags on every chunk

Instead of listing documents per user, store principals per chunk. At ingestion, each chunk gets an acl_principals tag copied from its document's ACL (group:grp-payments-eng). At query time, the query API resolves the user's principal set — user id, transitively expanded groups and the tenant-wide tag — from the identity provider through a short-TTL cache (~15 ms). Both indexes evaluate acl_principals ANY_OF principal_set inside the search, so the top 30 are drawn only from chunks this user can read.

Filtering inside an ANN search has a cost. When a user can read only a small fraction of the tenant, HNSW traversal keeps landing on ineligible neighbours, so recall drops and latency rises. The mitigations are filter-aware traversal, an exact scan over the eligible subset when it is small (a few thousand vectors scan in milliseconds), and per-tenant partitions that keep graphs small (Architecture Deep Dive → Authorization Internals).

Live re-check before generation

Chunk tags reflect the document ACL as of the last ingestion, and the principal set may be up to a cache TTL old. So after reranking, and before anything is placed in the prompt, the pipeline re-checks each shortlisted chunk's document against the authoritative document-ACL store and a fresh principal resolution (~20 ms for a handful of documents). Chunks that fail are dropped and the next reranked candidate takes their place until four survive. A timeout or a missing record counts as a deny. A revocation committed after this check still affects only the next request: the time-of-check-to-time-of-use (TOCTOU) window is bounded by the revocation budget, not zero.

Propagating permission changes

When a document's permissions change at the source, the connector emits an ACL-change event on a priority lane of the queue, with removals ahead of grants. The writer updates the document-ACL store and rewrites the acl_principals tags of that document's chunks — no re-parsing, no re-embedding; for a five-chunk document it is a small metadata update. Group-membership changes need no index write at all: the next query resolves principals afresh once the identity cache expires. Both paths are what the 60-second and 2-minute revocation targets in Chapter 04 measure.

What a denied user sees

"Nothing relevant exists" and "something relevant exists but you cannot read it" must look identical: the same 200 response and the same wording — "I don't have access to information that would answer that." Distinguishing them, by status code or wording, would itself reveal that a restricted document exists. Only the server-side trace records which case occurred.

What remains

Retrieval is now relevant and authorized, but still revision-blind: r2 and r3 of the policy are both readable, both relevant and both returned, and a deleted runbook stays retrievable until something removes it. That is Stage 6.

Stage 1 already stores revisions and the fenced writer already prevents stale writes. Stage 6 puts that lifecycle to work on the query path, so retrieval prefers the current revision, can still explain what changed, and forgets deleted content immediately.

The limit it addresses

Revision r2 of the policy says reconciliation runs manually at month end; r3 says it runs nightly. Both are relevant to a reconciliation question, both are readable by a payments engineer, so Stage 5 retrieves both, and the model receives two contradictory instructions with nothing saying which is in force. Meanwhile a runbook deleted at the source last week, RUN-5, is still in the index and still cited.

Revision states

Every chunk carries its revision's status as an index tag, updated by the writer whenever the document pointer moves. The default retrieval filter adds status == "current", so ordinary questions see only the policy in force.

Supersession at query time

Questions about change are the exception. When the rewrite step detects change or temporal intent ("what changed", "since last month", "previous version"), the filter also admits the immediately superseded revision of each matching document, and the context builder labels each chunk with its status and date. The model can then answer "r3 replaced the manual month-end process with a nightly automated run" and cite both, with r2 explicitly marked as superseded rather than presented as current. Older history stays out of retrieval unless explicitly requested.

Tombstones

When a document is deleted at the source, the connector emits a delete event carrying the source's delete version. The writer records a tombstone and immediately sets the document's chunks to status = "tombstoned", so the default filter excludes them on the very next query. Physical removal of chunk text and vectors follows within a purge deadline (24-72 hours), batched to keep index compaction cheap. The tombstone also records the delete version, so a stale upsert replayed after the delete is rejected by the same version fence instead of resurrecting the document.

Index lag and reconciliation

Index lag — source edit to retrievable — is measured per connector and alerted against the targets in Chapter 04. Events can still be lost (a webhook outage, a connector bug), so a nightly reconciliation sweep lists each source and compares document versions, ACL versions and deletions with the index, repairing any drift. Without it, a single lost delete event leaves a document retrievable forever.

Embedding-model migration

Upgrading the embedding model is the largest lifecycle event: old and new vectors are not comparable, so the corpus is re-embedded without ever serving mixed vectors. The new model's index is built in a shadow namespace. From the moment the backfill starts, every mutation — upsert, ACL change, supersession, tombstone — is dual-written to both namespaces. The backfill re-embeds only current, non-tombstoned revisions (750,000 chunks, hours of batched work). A parity check compares chunk counts, tags and retrieval quality on an evaluation set. Then an atomic alias flip points queries at the new namespace and switches the query embedder to the new model in the same step; the old namespace is kept briefly for rollback (Architecture Deep Dive → Embedding Batching and Migration; Trade-offs → Embedding Migration).

What remains

Retrieval now delivers four relevant, authorized, current chunks. Nothing yet ensures the answer the model writes from them is faithful, or that its citations are real. That is Stage 7.

Retrieval decides what the model may read; Stage 7 decides what the model may say. It frames the prompt so the model answers only from the evidence, validates every citation against that evidence, and abstains when the evidence is not enough.

The complete query path

1. Authenticate the user and fix the tenant from the token, never from the request body. 2. Rewrite follow-up questions into standalone queries and detect change intent (follow-up turns only). 3. Resolve principals and embed the query, in parallel. 4. Hybrid retrieval in the tenant's partition with the pushdown filter (principals, status). 5. Fuse with RRF. 6. Rerank the ~40 candidates. 7. Live re-check the top results, fail-closed, until four survive. 8. Assemble the context with citation markers. 9. Generate, streaming tokens. 10. Validate citations, or abstain. 11. Persist the trace.

Framing the prompt

Retrieved documents are written by many people and can contain text that looks like instructions ("ignore previous instructions and..."). The prompt treats source text as data inside delimiters, the model is never given tools or secrets that such text could exploit, and the final output is still subject to citation validation. These reduce prompt-injection risk; they do not eliminate it (Architecture Deep Dive → Prompt Construction).

Citation validation

A citation is only meaningful if it is checked. As the answer streams, a structural check confirms that every [n] refers to a chunk that was actually in this prompt — a marker pointing outside the context is dropped. After generation, an entailment check with a small model tests whether each cited chunk supports the sentence it is attached to; unsupported sentences are flagged, and the final done event carries only validated citations. Validation reduces unsupported claims; it cannot prove the absence of every subtle misstatement, which is why groundedness is also sampled offline (Chapter 14).

Abstention

If no chunk survives authorization, or the best reranker score falls below a calibrated threshold, the pipeline does not call the model to "try anyway": it returns the uniform "I don't have access to information that would answer that." response from Chapter 11. The model is also instructed to say when the sources it was given do not answer the question. It must never fall back on its parametric knowledge for company facts — a confident guess about internal policy is worse than no answer.

The query trace

Every request writes a trace: tenant, user, principal fingerprint, the retrieved, re-checked and cited chunk ids, the model versions and the timings. It is the audit record ("why did this user see this?"), the input to evaluation, and what makes a reported bad answer reproducible.

What remains

The pipeline is now correct for one request. At 150-200 queries per second across 500 tenants it must also be affordable, isolated between tenants under load, and correct when a dependency fails. That is Stage 8.

The final stage keeps every guarantee from the previous stages intact under load and partial failure: caches that cannot leak across users, capacity that one tenant cannot exhaust, degraded modes that never skip an authorization check, and quality that is measured rather than assumed.

Caching without leaking

Two users in the same tenant asking the identical question may be entitled to different answers. A cache keyed on the question text alone would hand a payments engineer's answer to a sales user. Only one thing in the pipeline is identity-independent: the query embedding, which depends on the text and the model alone. Retrieval results under pushdown, reranked lists and generated answers all depend on who is asking, so their keys must include the requester's entitlement and the corpus state (Architecture Deep Dive → Authorization-Safe Caching; Trade-offs → Cache Keys vs. Leakage).

The TTL bound matters because a group removal changes the principal set only when the identity cache refreshes; a cached answer must not outlive that budget. In practice per-identity answer caches have low hit rates, so many deployments cache only embeddings and retrieval results and accept the generation cost.

Scaling the query path

Retrieval scales by replicating each tenant's partitions: ~5 GB of vectors fits in memory per replica, and replicas add throughput and availability. The LLM tier is the bottleneck (~1,000 concurrent generations at peak, Chapter 05), so it gets admission control, per-tenant token quotas and a queue with a time-to-first-token deadline. Tenant partitions bound data exposure but not resource contention: a large tenant's spike can still exhaust shared rerankers or LLM quota, so noisy-neighbour protection comes from per-tenant rate limits and, for the largest tenants, reserved capacity (Trade-offs → Tenant Partitioning).

Scaling ingestion

Ingestion runs on a separate worker pool fed by the queue, scaled on queue depth, with embedding requests batched for throughput. Bulk events — an initial sync, a re-embedding migration — run at lower priority than incremental updates, and ACL removals always go first. A backlog shows up as index lag, never as query latency (Trade-offs → Sync vs. Async Ingestion).

Failure boundaries

The rule: degrade relevance or freshness if necessary, never authorization. Each dependency has a defined degraded mode:

Measuring quality

Retrieval quality and generation quality fail independently, so they are measured separately on a labelled evaluation set, re-run on every change to chunking, models or prompts, and complemented by production feedback.

Operationally, the system also tracks index lag, revocation-propagation time, abstention rate and per-stage latency against their budgets (Architecture Deep Dive → Observability).

The system exposes two very different surfaces: a trusted, service-to-service surface for connectors and administrators, and a user-facing query surface. Keeping them separate — different authentication, different network exposure — is itself an authorization control.

Connector setup (admin, internal)

Ingestion (connector-authenticated, idempotent)

Permission change (connector-authenticated)

Querying (user-authenticated, streamed)

A denied or unanswerable query

Feedback and lifecycle operations

Error semantics

Each failure gets a distinct, explicit outcome because each needs a different operator response: a duplicate ingestion returns the original accepted response; a stale revision returns 409 STALE_REVISION_REJECTED; invalid connector credentials return 401 on the internal surface; a query exceeding the tenant's quota returns 429 with a retry hint. None of these distinctions is ever exposed on the user surface in a way that depends on restricted content.

The RAG architecture diagram below shows the complete design assembled from the eight stages: an asynchronous ingestion path that keeps a current, chunked, permission-tagged, versioned copy of every source in tenant-partitioned lexical and vector indexes; a synchronous query path that retrieves, authorizes, reranks and generates a grounded, cited answer; and the stores and checks that connect them. Every component exists because a specific earlier stage needed it.

Component responsibilities

Request flow: ingestion

A source edit reaches a connector by webhook or poll. The connector fetches the new revision's content and ACL and enqueues one event. A worker parses and chunks the text, embeds the chunks in a batch, writes chunks, vectors and tags to the tenant's partitions, applies the fenced pointer update (which marks the previous revision superseded) and advances the sync cursor. A permission change takes the priority lane and rewrites tags only; a deletion writes a tombstone. Typical end-to-end lag: minutes.

Request flow: query

For a payments engineer, the running question returns an answer grounded in r3, contrasted with the labelled superseded r2 and citing RUN-8 for the steps. For a user only in grp-sales, the pushdown filter admits no payments chunks, nothing passes the relevance threshold, and the uniform denial is returned without calling the model.

How the architecture evolved

A trustworthy design says what it does not guarantee. RAG invites overconfidence because a fluent, cited answer looks certain whether or not the guarantees behind it hold.

What this design guarantees

Structural tenant isolation: a query searches only its tenant's partitions. Fail-closed authorization on every query: pushdown on current principals, plus a live re-check before generation. Idempotent, version-fenced ingestion: a duplicate event has no effect and a stale one cannot resurrect a superseded or deleted revision. Validated citations: every returned citation refers to a chunk that was in the prompt and passed the support check. An auditable trace for every answer.

What it does not guarantee

Relevant retrieval does not guarantee a correct answer. The right chunk can be retrieved and still be misstated during generation; that is why citation validation and groundedness evaluation exist as separate checks.

A grounded answer is not necessarily a current one. An answer can faithfully restate a superseded revision if a supersession signal was missed. Faithful to its source and using the right source are independent properties.

Content freshness and permission freshness are separate pipelines. A document can be current while its ACL change is still propagating, or the reverse; each needs its own lag monitoring.

No zero-hallucination guarantee. Validation reduces unsupported claims but cannot prove every sentence faithful. No answers from sources that were never connected, or from a connector that silently stopped syncing — the honest response is abstention. No invented lineage: if the source never records that one document replaces another, the system cannot know.