Enterprise RAG Knowledge Assistant — Trade-offs and Decisions

Employees ask in their own words ("do we still reconcile payments by hand at month-end?") about text that says "automated nightly run with an exception queue", and they also ask about RUN-8, which must never be answered from RUN-5. Both kinds of miss produce wrong answers, and retrieval has ~80 ms of the budget.

Selection criteria. Recall@k on paraphrase and identifier questions separately; robustness when one scorer changes; latency; the cost of keeping two representations consistent.

When another option wins. BM25 only for identifier-lookup tools (ticket or part-number search) where users type the document's own vocabulary. Dense only for narrative corpora without codes, or for cross-lingual search, where queries and documents rarely share terms. Learned sparse retrieval (e.g. SPLADE) when one index matters more than an extra ingestion-time model; weighted score blending only with stable, normalised scorers.

Remaining risk. A standard tokenizer splits RUN-8 into run and 8, silently erasing the identifier advantage unless the analyzer preserves identifiers (Architecture Deep Dive, Component 04). RRF ignores score magnitude, so a clear single-list winner can lose to consensus candidates; the reranker corrects most such cases.

Chunks are the unit of retrieval, citation and permission tagging, and only four fit the context (~1,600 tokens). Sources differ in shape — headed wiki policies, drive documents with tables, chat threads without headings — and a boundary through the sentence that states a rule makes both halves useless.

Selection criteria. Recall@k per source type; predictable sizes for capacity and token budgets; ingestion cost; whether a citation points at a meaningful passage.

When another option wins. Fixed-size windows for a deadline-bound bulk import, with a planned re-chunk afterwards. Parent-child when answers need whole sections (contracts, procedures) and the context budget can afford it. Semantic chunking for long transcripts with no structure at all.

Remaining risk. Any chunking change is corpus-wide: it re-chunks and re-embeds all 750,000 chunks and must be published as a staged index version (Decision 14). Boundary regressions show up only as recall@k drops per source type (Architecture Deep Dive, Component 10), not as errors.

Authorization is zero-tolerance and fail-closed: no answer, citation or paraphrase may reveal content the requester cannot read now. Users differ widely in what they can read (a user only in grp-sales asking a payments question can read few of the relevant chunks), and permissions can change while a request is in flight.

Selection criteria. Whether exposure is bounded and measurable; recall for users with narrow access; behaviour when a dependency times out; added latency.

When another option wins. Never for access-controlled content: no workload makes output filtering acceptable, and no workload with per-document permissions should drop the re-check. Only the surroundings vary: a corpus every tenant member may read in full (a public handbook) has no per-document ACL, so tenant partitioning alone suffices.

Remaining risk. A revocation committed after the re-check affects only the next request; a group removal can take up to the 60 s cache TTL (Decision 05). Filtered HNSW loses recall when a user can read only a few percent of the tenant; the planner offsets it with exact scans and a wider search (Architecture Deep Dive, Component 06).

Index lag must stay within 5 minutes p95 for small edits and 15 minutes for large documents; document-ACL removals must apply within 2 minutes. Source APIs are rate-limited, webhooks are best-effort, and a deletion that produces no event must still be found.

Selection criteria. Lag and revocation budgets; each source's delivery guarantees; API quotas; whether deletions and ACL edits are observable.

When another option wins. Polling alone for sources without webhooks or with low churn. A full recrawl as the primary mechanism for small sources with no change API at all. Dropping the sweep never pays: it is cheap and is the only layer that finds silent deletions.

Remaining risk. A source whose change feed omits permission edits can surface an ACL removal only through the sweep, which is outside the 2-minute budget; such connectors need a dedicated permissions poll or an explicit exception. Polls compete with user traffic for the source's rate limit during bulk edits.

Group membership lives in the identity provider (IdP) and is resolved per query. A group removal must take effect within 60 s, principal resolution has ~15 ms of the budget, and an IdP outage must neither leak content nor take down a 99.9%-available query path.

Selection criteria. The revocation budget; added latency; coupling of query availability to the IdP; behaviour when the IdP fails, which must be closed.

When another option wins. A live check per query for a regulated tenant whose contract requires removal within seconds — applied to that tenant only, accepting its latency and availability cost. A longer TTL only if the product explicitly relaxes the revocation budget; the TTL follows the budget, never the reverse.

Remaining risk. Up to 60 s of exposure after a group removal when no event arrives. During an IdP outage users lose group-granted content, which shows up as a quality drop rather than a leak. The revocation-lag canary in Architecture Deep Dive, Component 10 verifies the bound continuously.

About 500 tenants of very different sizes share the platform; the largest has 750,000 chunks (~5 GB of vectors). Cross-tenant reads must be impossible, and one tenant's spike must not degrade others.

Selection criteria. Isolation enforced by structure rather than convention; blast radius of a bug; per-tenant operations; compute isolation; overhead for the long tail of small tenants.

When another option wins. Only the granularity changes, never logical isolation: thousands of tiny tenants fit as namespaces on shared clusters, provided the engine enforces the namespace rather than a query predicate. A single-tenant deployment makes the question moot. A shared filterable index is not an acceptable option for mutually distrusting tenants.

Remaining risk. Partitions do not stop contention for shared rerankers or LLM quota; every shared resource needs its own per-tenant limit. Isolation also depends on deriving the tenant only from the authenticated token, never from a request parameter.

Per tenant partition, retrieval needs BM25 with custom analyzers, permission-filtered ANN that keeps recall under selective filters, cheap acl_principals/status tag updates and an atomic alias switch — with BM25 and vectors never disagreeing about which chunks exist.

Selection criteria. A checklist: filtered ANN with an exact-scan fallback; tag-only updates; identifier-preserving analyzers; atomic aliases; per-tenant namespaces; then operating capacity and cost.

When another option wins. Self-hosting when managed pricing across the platform exceeds the cost of an infrastructure team, or when a checklist item (a custom analyzer, filter-planner control) is not exposed. Two stores when a required vector capability exists only in a dedicated vector database — accepting the commit point and reconciliation job that come with it.

Remaining risk. In segment-based engines a tag update re-indexes the whole record, vector included, so a bulk ACL change is real write load (Architecture Deep Dive, Component 06). Moving between engines later is a corpus-wide migration published like Decision 14.

Fusion yields ~40 candidates, but only four fit in the context. First-pass scorers judge the question and each chunk separately, so a glossary entry that mentions reconciliation can rank next to the passage describing the change. A cross-encoder costs ~120 ms per query.

Selection criteria. The measured gap between fusion order and reranked order on the evaluation set (nDCG and context precision per question type, plus groundedness of the resulting answers); the latency budget; reranker cost relative to generation, which dominates per-query cost.

When another option wins. Conditional reranking for very high-volume tenants once reranker spend becomes material and the threshold is validated on that tenant's queries. Fusion-only as the normal mode when the evaluation gap is negligible (a narrow identifier-lookup corpus). An LLM as reranker only where its flexibility is worth a slower, costlier judgement per pair.

Remaining risk. A reranker version change can bury answers without any error; it is gated on the evaluation set like any model change (Architecture Deep Dive, Component 10). Degraded mode is lower precision by design, so its duration is an alerting metric.

Every one of 750,000 chunks has a 1,536-dimension vector; steady churn is only 6,000 embeddings a day, but a model upgrade changes every vector, and vectors from different models are not comparable. Embedding the corpus at query time is not an option at this size, so the real choice is the migration strategy.

Selection criteria. Correctness (one vector space per search); rollback speed; temporary storage and embedding cost; how long the migration lasts while 1,200 revisions a day keep arriving.

When another option wins. Pinning the model is right until evaluation shows a meaningful gain or the provider announces retirement. Querying both models and fusing during the transition suits a cautious rollout, at two embeddings and two ANN searches per query until the cutover.

Remaining risk. Events that arrive during the backfill must reach both namespaces through the same version-fenced writer, or the new namespace is born stale. The query embedder and the index alias must switch together; a mismatch is the in-place failure again.

Two users in the same tenant asking the identical question may be entitled to different answers. Any cache placed after authorization must therefore reuse a value only for the same entitlement and corpus state, within the 60 s revocation budget. Per-identity answer caches tend to have low hit rates.

Selection criteria. Leakage must be impossible by construction; then hit rate, staleness bounds and the cost of validating a hit.

When another option wins. Skip answer caching entirely when principal sets are so varied that fingerprints rarely repeat — the embedding cache then captures most of the saving. Hit rates rise naturally where entitlements are uniform (most users share one fingerprint), and the same keys stay safe.

Remaining risk. The fingerprint must cover every input to authorization; adding a new entitlement dimension (say, a clearance attribute) without adding it to the key reintroduces the leak. A cached result cannot see a newly published document that would now outrank its evidence; the TTL bounds that.

The corpus is ~225 million tokens (150,000 documents × 1,500 tokens), and each query is budgeted at ~2,200 input tokens, LLM tokens being the dominant cost. Every passage given to the model must also be one the user may read.

Selection criteria. Token cost at 4,400 queries a day per tenant; time to first token; groundedness; citation granularity.

When another option wins. Long context wins for single-document "chat with this file" features and for small corpora of a few dozen documents with uniform access, where the retrieval machinery costs more than it saves.

Remaining risk. A question whose answer spans more than four chunks gets a partial answer; the support check and abstention keep it honest, and recall@k on multi-chunk questions is tracked separately (Architecture Deep Dive, Component 10).

Generation handles ~2,200 input and 350 output tokens per query with ~1,000 concurrent generations at platform peak. Retrieved text is untrusted, so the model must treat it as data, not instructions, and groundedness must stay at or above 95%.

Selection criteria. Measured groundedness and citation precision; an injection test suite; cost per query; time to first token under peak load.

When another option wins. A smaller generator for a tenant once its groundedness, citation precision and injection-suite results on that tenant's evaluation set meet the bar — a measured, per-deployment decision. Routing pays off at very high volume with many simple lookups.

Remaining risk. No model size makes prompt injection impossible. Structural separation of instructions and evidence, citation validation and a generator without tool privileges must hold whichever model runs (Architecture Deep Dive, Component 08).

Connectors deliver at-least-once and in bursts: the initial sync is 750,000 chunks, and bulk edits arrive in waves. The query path must stay 99.9% available whatever ingestion is doing, and freshness is a stated lag budget rather than "immediately".

Selection criteria. Isolation of query availability from ingestion; behaviour under bursts; whether freshness is a monitored number; failure handling for at-least-once delivery.

When another option wins. A synchronous confirmation only as an admin "publish and verify" tool that polls revision status with a timeout — never by making the ingest path itself blocking. Small single-tenant systems with trivial volume can afford synchronous writes.

Remaining risk. A large backlog consumes the lag budget silently unless lag is measured per tenant and per lane; bootstrap syncs should run in their own lane so they cannot delay incremental edits.

Two kinds of change reach the index. About 1,200 revisions a day each touch one document; an embedding-model, chunking or analyzer change touches all 750,000 chunks. Queries must never see a half-applied corpus-wide change.

Selection criteria. Whether the change touches one document or the whole corpus; consistency of what a query sees; rollback time; temporary cost.

When another option wins. Never apply a corpus-wide change in place to an index that is serving queries. In-place is acceptable for such a change only when the index is not serving — a new tenant before launch, or a planned outage short enough to fit the freshness budget.

Remaining risk. Revisions arriving during the build must be dual-written through the fenced writer; the bake period must be long enough to catch quality regressions before the old version is deleted.

The 14 decisions fall into two groups. Some protect invariants and have no legitimate alternative for access-controlled enterprise content; the rest follow from this workload's numbers and should be revisited when those numbers change.