Enterprise RAG Knowledge Assistant — Interview Perspective

"Design an AI assistant over our internal documents" is underspecified in ways that change the architecture. The first five minutes decide whether the rest of the hour is spent on the right problem.

Start by drawing the boundary: this system ingests already-connected sources, retrieves evidence for a question, and generates a cited answer from that evidence. It does not browse the open web, author documents, take autonomous actions, or fine-tune a model on the corpus. Candidates who start describing agent loops or training pipelines are solving a larger, different problem.

The questions that change the design

“"I'll scope this as a multi-tenant assistant over each customer's wiki, drive and chat. Access control is document-level, enforced on every query against current group membership. Every answer is grounded and cited, and the system abstains when the evidence is insufficient. Freshness and revocation are stated budgets — minutes for content, about a minute for group removal — not 'instant'. Authorization and tenant isolation are zero-tolerance invariants, not trade-offs like latency or cost. Does that match what you want me to focus on?"”

The numbers matter less than the three decisions they force. State assumptions, round aggressively, and land on the conclusions.

What to say. "The largest tenant's index is about 5 GB of vectors, which fits in memory on one node — I don't need to shard it for size. Partitions exist for isolation, replicas for throughput. The bottleneck is the LLM tier: about a thousand generations in flight at peak, so provider quota or GPU capacity sets the ceiling, and it needs admission control and per-tenant quotas. Every context token is paid on every query, so context size is the main cost lever — which is why I'll spend a reranker to keep the prompt to four chunks. Ingestion is a trickle except for initial syncs and model migrations, so it runs asynchronously on its own pool." (Complete Design, Ch. 05)

Grow the design one stage at a time, each motivated by a concrete limitation of the stage before. For each stage: what to draw, what to say, the trade-off to name, and the follow-up an interviewer is likely to ask. The board ends at the production architecture — but you arrive there, rather than drawing it first.

A running example keeps every stage concrete: a payment-reconciliation policy POL-17 with an older revision r2 (manual month-end) and a current revision r3 (automated nightly), a runbook RUN-8, a group grp-payments-eng that may read both, and a user only in grp-sales who may not. The question is "What changed in our payment reconciliation process last month?"

Stage 0 — The naive baseline

Draw. Sources copied nightly into a documents table; a chat app runs a keyword LIKE search and sends the matching whole documents to an LLM.

Say. Give it one sentence of credit — it works for a demo — then list where it breaks, because each break becomes a later stage: vocabulary mismatch (the user says "reconciliation", the runbook says "matching job"); whole documents blow the context budget and nothing ranks them; the copy is stale and revision-blind, so r2 and r3 look equally valid; there is no permission check on the answer path; and nothing is cited.

Trade-off to name. Simplicity against every requirement. The point of Stage 0 is not to defend it but to produce the roadmap.

Likely follow-up. "Why not fine-tune a model on the documents, or put everything in a long context window?" — Fine-tuning cannot be updated per edit, cannot forget a deleted document, cannot respect per-user permissions and cannot cite. Long context does not scale to 150,000 documents, multiplies cost on every query, and still lets the model read content the user may not see. Retrieval selects a small, authorized, current, citable subset per question.

Stage 1 — Ingestion pipeline and data model

Draw. Connectors (content and ACLs in the same pull) → durable queue → parse and chunk → index writer → lexical (BM25) index over chunks and a metadata store of documents, revisions and ACLs.

Say. "Delivery is at-least-once and out of order, so the writer is idempotent on (doc_id, revision_id) and version-fenced: it moves the current pointer only if the incoming source version is newer and the document is not tombstoned. The sync cursor advances last. Chunk ids are scoped to a revision, so r2 and r3 are different rows and history survives."

Trade-off to name. Webhooks for latency, polling for coverage, nightly reconciliation as the backstop; asynchronous ingestion so a backlog shows up as index lag, never as query latency.

Likely follow-up. "A replay of r2 arrives after r3 is live. What happens?" — The conditional update ... WHERE current_source_version < :v AND tombstoned_at IS NULL matches zero rows; the event is acknowledged without writing and recorded as 409 STALE_REVISION_REJECTED.

Stage 2 — Embeddings and vector retrieval

Draw. Embedder (batched at ingestion, one call per query) and an HNSW vector index of 1,536-dimensional vectors beside the BM25 index.

Say. "Embeddings fix vocabulary mismatch. Chunk size now matters: ~400 tokens with ~100 overlap keeps a passage focused without splitting the key sentence. The embedding model version is stored per chunk, because vectors from different models are not comparable."

Trade-off to name. Chunk size: smaller chunks retrieve precisely but lose context; larger ones dilute the vector and cost more prompt tokens.

Likely follow-up. "How big is the vector index, and does it need sharding?" — 750,000 chunks × 6 KB ≈ 4.6 GB, about 5 GB with graph links — it fits in memory; replicate for throughput, partition for isolation.

Stage 3 — Hybrid retrieval

Draw. The retriever fans out to BM25 and ANN in parallel (top 30 each) and fuses them with reciprocal rank fusion into ~40 candidates. There is no separate "hybrid index" — two indexes, one retriever.

Say. "Vectors blur exact identifiers: RUN-8 and RUN-5 embed almost identically. BM25 catches identifiers and rare terms; vectors catch paraphrase. RRF scores by rank, 1/(k + rank) with k ≈ 60, so the two scorers never need normalising."

Trade-off to name. RRF versus weighted score blending (needs normalisation, drifts when a scorer changes); one engine storing text, vector and tags together versus two stores that need a shared commit point and reconciliation to avoid drift.

Likely follow-up. "Why not just add the BM25 score and the cosine similarity?" — They are on different, corpus-dependent scales; a weighted sum needs per-corpus normalisation and re-tuning whenever either scorer changes. Rank fusion is robust by default.

Stage 4 — Reranking and the context budget

Draw. A cross-encoder reranker over the ~40 fused candidates, and a context builder that keeps the best four chunks (~1,600 tokens), de-duplicates overlap, caps chunks per document and adds citation markers.

Say. "Forty candidates are ~16,000 tokens — ten times the budget — and first-pass scores are too coarse to choose the best four. A cross-encoder reads query and passage together, ~120 ms over 40 pairs."

Trade-off to name. ~120 ms and a model per query against precision; conditional reranking when one candidate clearly dominates; context size against cost and quality.

Likely follow-up. "Why not send twenty chunks and let the model sort it out?" — Each chunk adds ~400 input tokens to every query (~18% more input at this size), marginal passages distract the model, and facts in the middle of long contexts are used less reliably. More context is not monotonically better.

Stage 5 — Authorization as a retrieval constraint

Draw. Per-tenant partitions of both indexes; acl_principals tags on every chunk; an Identity & ACL store; principal resolution in the query API; the filter inside both searches; a live re-check after reranking; a priority lane in the queue for ACL changes.

Say. Walk the rejected options in order — it is the strongest part of the answer. "Tenants get a partition, not a WHERE clause, so a forgotten predicate cannot cross tenants. For document permissions, post-filtering the top 30 starves restricted users: a sales user keeps two or zero chunks even when a readable answer exists at rank 200. Filtering by doc_id IN (list) needs a per-user list of tens of thousands of ids rebuilt on every change. Instead, each chunk carries its document's principals, the query resolves the user's small principal set — user id, expanded groups, a tenant-wide tag — and both indexes evaluate acl_principals ANY_OF principal_set inside the search. Then, before anything enters the prompt, a live re-check against the authoritative ACL store, fail-closed. A denied user gets the same 200 response as 'nothing found': I don't have access to information that would answer that."

Trade-off to name. Filtered HNSW loses recall when a user can read only a small fraction of the tenant — mitigated by filter-aware traversal, an exact scan of a small eligible set, and per-tenant partitions. The TOCTOU window is bounded by the revocation budget, not zero.

Likely follow-up. "A user is removed from a group. When do they lose access?" — Group membership is resolved per query through a cache with a TTL inside the 60-second budget, and the live re-check resolves it afresh, so the next request is denied. A document-level ACL change travels the priority lane and rewrites only that document's chunk tags — no re-embedding — within 2 minutes p95. A request already past its re-check can still complete: that is the bounded TOCTOU window.

Stage 6 — Freshness and lifecycle

Draw. A status tag in the retrieval filter, tombstones, a nightly reconciliation sweep and a purge job.

Say. "Both policy revisions are readable and relevant, so Stage 5 returns both. The default filter adds status = current. When the question has change intent — 'what changed', 'since last month' — the filter also admits the immediately superseded revision, labelled, so the model can contrast r3 with r2 and cite both. A deletion writes a tombstone that excludes the document on the next query; physical purge follows within 24-72 hours."

Trade-off to name. Keeping history for "what changed" questions against confusing the model with outdated text; purge speed against index-compaction cost.

Likely follow-up. "How do you upgrade the embedding model without downtime?" — Build a shadow namespace; dual-write every mutation — upserts, ACL changes, supersessions, tombstones — from the start; backfill only revisions that are still current and not tombstoned; run a parity check; then flip one alias that switches the index and the query-embedding model together. Keep the old namespace briefly for rollback.

Stage 7 — Grounded generation

Draw. A framed prompt, a citation validator after the LLM, the abstention path, and the query trace store.

Say. "Instructions, labelled sources and the question go in separate delimited sections; source text is data, never instructions. As the answer streams, a structural check confirms each [n] refers to a chunk in the prompt; afterwards an entailment check tests that each cited chunk supports its sentence, and the final event carries only validated citations. Below a calibrated relevance threshold the pipeline abstains without calling the model. Every request writes a trace of retrieved, re-checked and cited chunk ids."

Trade-off to name. Streaming latency against validating everything before display; abstaining too often (unhelpful) against too rarely (fabrication).

Likely follow-up. "What stops a document from hijacking the model with planted instructions?" — Source-as-data framing, no tools or secrets for the generator, and citation validation reduce the risk; authorization before the prompt limits the blast radius, because the context holds only what this user may read. It is reduced, not eliminated.

Stage 8 — Caching, scaling, failures, evaluation

Draw. The principal-scoped cache beside the query API; per-tenant quotas and admission control in front of the LLM tier; the degraded mode next to each dependency.

Say. "Only the query embedding is identity-independent. An answer cache key is tenant + normalized query + principal fingerprint + corpus version, with a TTL inside the revocation budget — and per-identity answer caches hit rarely, so caching embeddings and retrieval results is often enough. The LLM tier gets admission control and per-tenant token quotas; partitions isolate data but not compute. Every dependency has a degraded mode that keeps authorization intact. Quality is measured per stage: recall@k for retrieval, groundedness and citation precision for generation, explicit authorization tests."

Trade-off to name. Answer-cache hit rate against leak risk; reserved capacity for large tenants against utilisation.

Likely follow-up. "The identity provider is down. What do you serve?" — Fail closed: answer only from tenant-wide chunks, or refuse. Never an expired principal set, never a skipped re-check. Likewise: LLM down → the authorized, ranked source list; reranker down → fusion order with a tighter context; vector index down → lexical-only, flagged.

Around minute 35 the interviewer will pick one mechanism — or ask you to. Offer the one where this system differs most from generic search: authorization. Have two others ready.

Whichever you pick, structure it the same way: the invariant it protects, the simplest approach and where it breaks, the mechanism, its cost in latency or complexity, and its residual failure mode. Ending on the residual — "the TOCTOU window is bounded, not zero", "validation reduces unsupported claims, it cannot prove their absence" — is what distinguishes depth from recitation.

Some interviews replace or follow the design with an incident: "a customer reports X — walk me through it." The pattern is the same each time: scope, read the system's own evidence, localise the stage, fix the instance, then remove the cause.

Prompt A — "The assistant cited a runbook that was deleted two weeks ago"

1. Scope before theorising: which tenant, which document, reproducible now or once, when was it deleted versus when was the question asked.

2. Read the trace: if the deleted document's chunk ids are in the retrieved and cited ids, the model was given it as evidence — this is an index problem, not a hallucination.

3. Check the lifecycle record: a tombstone exists and is correctly dated, so the document should have been excluded on the next query. Are its chunks still in the indexes, and with what status?

4. Find the write: an index write after the tombstone, with no source version, means something bypassed the index writer — the fence was never consulted.

5. Fix the instance: re-apply the tombstone through the writer and disable the tool that wrote directly. Remove the cause: only the writer may write to the indexes; reconciliation checks tombstoned-but-retrievable documents frequently. This is Scenario 03.

Prompt B — "Two employees asked the same question; one saw content they should not have"

Treat it as a security incident first. From the trace: was the response a cache hit? Compare the stored and requester principal fingerprints (Scenario 08). If not, did the pushdown filter admit the chunk — were its tags stale because an ACL change was still in the priority lane? Did the live re-check run and pass, and against what ACL-store state? Was the revocation inside the stated budget (Scenario 07)? Each answer localises the failure to one component; the trace then lists every other affected request.

Prompt C — "Answers got worse after Tuesday's deploy, but nothing is erroring"

Split quality by stage before guessing. Recall@k down means retrieval (a chunking change, an index setting, an embedding model); recall flat but groundedness or citation precision down means generation or context assembly; abstention rate moving means the threshold or the reranker. A platform-wide shape points at infrastructure, a single-tenant shape at content (Scenarios 05, 10, 12).

These probe whether the invariants are enforced or merely claimed. Expect them in the last fifteen minutes, or as pushback on a core answer.

The pass/fail line is rarely knowledge of RAG. It is whether authorization gates what the model is shown, or merely filters what it says.

Questions beyond the walkthrough, grouped by theme, each with an answer outline and a follow-up.

Retrieval and generation

Operations and extensibility

Five minutes before the interview, rehearse the sequence and the framings — not the diagram.