Designing ChatGPT / an AI Assistant

We need to design a ChatGPT-style assistant that supports multi-turn chat, token streaming, file uploads, retrieval over documents, durable memory, and tool-assisted workflows while staying fast, reliable, and economically viable at internet scale.

The v1 mental model is deceptively simple: send the prompt to a model and print the answer. The real product problem begins immediately after that first demo works. Users expect the assistant to remember the conversation, cite uploaded material, call tools, survive provider incidents, and return answers in seconds rather than tens of seconds.

Why this system is interesting

An AI assistant combines classic distributed-systems concerns with model-specific constraints. You still design APIs, storage, caching, queues, and failover, but now you also manage context windows, token costs, prompt assembly, provider routing, and unsafe tool execution. The architecture has to reason about both software bottlenecks and model bottlenecks.

Five planes

It helps to decompose the system into five planes that recur throughout this module: the conversation plane (turns, persistence, context, streaming), the knowledge plane (files, ingestion, retrieval, provenance), the memory plane (durable user/workspace context across sessions), the action plane (tools and agents), and the model platform plane (gateway, routing, cost, reliability). Most architectural questions in this design map cleanly onto one of these five.

Functional requirements define the behaviors the user directly experiences: conversation creation, messaging, streaming, document understanding, memory, and actions through tools.

Non-functional requirements are what force the architecture away from a toy demo and toward a production system.

Scoping matters because an AI assistant can easily expand into 'everything an agent might ever do.' A disciplined interview answer keeps the core crisp.

In scope

Multi-turn chat, conversation persistence, SSE token streaming, file upload and async processing, embeddings, vector retrieval, rolling conversation summaries, long-term memory, tool calling, agent-style multi-step workflows, provider routing, rate limiting, and production reliability.

Out of scope

Model training or fine-tuning infrastructure, GPU cluster internals, multimodal generation beyond file understanding, enterprise tenant administration, billing ledgers, deep safety policy systems, and custom model architecture research. Those are real systems, but they are not required to explain the assistant architecture cleanly.

This module frames the exercise as a ChatGPT-style / production AI assistant architecture — a generic, well-known product shape used as a familiar reference point. It is not a claim about OpenAI's actual private ChatGPT production architecture. Where this module names technologies such as Postgres, Redis, S3, or HNSW, or a specific gateway design or memory approach, it is describing one reasonable architecture for this design exercise, not ChatGPT's verified internals.

Capacity planning for an AI assistant is dominated by message throughput, token volume, and inference cost — not just row counts in a database. Every number below is an illustrative design assumption, not a claim about any real product's traffic or a current provider price.

Traffic assumptions

For an illustrative large-scale design, assume 10 million DAU. If each active user sends 8 messages/day, that is 80 million messages/day. 80M ÷ 86,400 ≈ 926 messages/sec average. Peak traffic is highly bursty; using a 12× peak factor gives roughly 11,000 messages/sec peak on the write path into the chat system.

Token math

Assume prompt construction after summaries and retrieval averages 1,200 input tokens and responses average 250 output tokens. Daily volume is therefore 96B input tokens/day and 20B output tokens/day. Without summaries and budget enforcement, that input number grows much faster than user traffic because every extra turn drags more history behind it.

Inference concurrency

Message rate alone understates operational load. Estimate concurrent generations ≈ request rate × average generation duration — if the average generation takes several seconds, thousands of messages/sec can mean tens of thousands of concurrent in-flight generations, which is what actually sizes provider concurrency limits and gateway capacity, often more than raw API QPS does.

Streaming connections

Estimate concurrent SSE streams as a separate figure from request rate — a stream stays open for the duration of generation, so the number of simultaneously open streams tracks concurrent generations plus any idle-but-connected clients, not the per-second message rate.

Agent amplification

One user turn is not one inference call once tools and agents are involved — it may produce multiple model calls, multiple tool calls, retrieval calls, and verifier/reranker calls. Define an illustrative amplification factor (for example, "assume agent-eligible turns average 3–5 model calls") rather than assuming a flat one-message-to-one-inference ratio; this factor can dominate cost and concurrency for agent-heavy products.

Retrieval and index capacity

Estimate chunks/day, total vectors, embedding throughput, re-index load from document updates, and query-time retrieval QPS as their own capacity line, separate from chat message throughput — a retrieval-heavy product can be bottlenecked on embedding/indexing capacity well before chat API capacity becomes the constraint.

Tool workload

A tool-heavy product may end up limited by external API rate limits rather than by inference capacity — a coding agent calling a repository API, or a research agent calling a search API, can hit a ceiling that has nothing to do with LLM throughput.

Illustrative inference cost

Treat any dollar figures as illustrative and dated, expressed parametrically where possible: daily inference cost = input_tokens × input_price + output_tokens × output_price, summed per routing tier. As one illustrative scenario: suppose routing sends 60% of traffic to a cheap/fast tier, 30% to a high-capability reasoning tier, and 10% to a coding-specialized tier, each with its own assumed per-token price; multiplying through the token volumes above yields a blended daily spend in the hundreds of thousands of dollars for a product at this scale — the exact figure is far less important than the shape of the calculation and the fact that routing mix materially changes it.

Storage and ingestion

If each persisted message row, metadata included, averages 2 KB, then 80M messages/day adds about 160 GB/day to hot conversation storage. Keeping 30 days hot requires ~4.8 TB plus indexes and replicas. For file workflows, if 2% of the 10M DAU each upload one file per day, that yields 200k files/day at 2 MB average, object storage ingests 400 GB/day raw. If each file becomes 50 chunks, the system creates 10M chunks/day for embeddings and vector indexing — and snapshot/history write pressure scales with retention policy and compaction cadence on top of that.

The public API should separate durable resource creation from streaming delivery. REST is fine for metadata; SSE is ideal for the live response channel — and message submission needs to establish a stable logical turn identity, not just accept text.

Cancellation

POST /turns/{turnId}/cancel is best-effort: it stops the downstream provider stream, stops any further agent steps, and cancels cancellable tools — it cannot undo an external side effect that already completed. Cancellation propagates through the turn's current state rather than assuming the client's disconnect alone is sufficient signal (see Token Streaming in Architecture Deep Dive).

Retry and regenerate

A retry request must specify what is actually being retried: reconnecting to the same logical turn/provider attempt via the idempotency key, starting a new assistant attempt for the same user turn, or creating an explicit regenerate branch. Conflating these produces ambiguous duplicate state — see Turn Idempotency and Concurrent Conversation Turns in Architecture Deep Dive.

Streaming contract

POST /conversations/{conversationId}/messages persists the new user turn first, establishes the logical turn, then opens an SSE stream for incremental delivery. Uploads use a pre-signed direct-to-object-storage flow so the app tier does not proxy large file bodies.

The data model splits durable conversation state, ephemeral acceleration state, retrieval artifacts, and durable memory. Each has a different access pattern and should live in the store that matches it.

Conversation Store (Postgres)

Redis acceleration state

Retrieval and memory artifacts

The Vector Database stores the embedding vectors plus chunk IDs and namespaces, keyed by the same index_generation used on the metadata row so the two can be reconciled if they ever drift (see Vector/Metadata Index Consistency in Architecture Deep Dive). Postgres remains the durable metadata store; Redis remains the speed layer. Keeping those boundaries clear makes recovery, auditing, and migrations much easier. source_location and permission_ref on each chunk are what make retrieval provenance and authorization checks possible at query time.

The best way to explain this system is to evolve it live. Each new component appears only when the previous stage demonstrably breaks.

Stages 1–5 — from demo to usable chat

1. Basic ChatGPT: User → Chat Service → LLM → Response. 2. Conversation History: persist turns in Postgres so the chat survives refresh. 3. Context Window Challenges: naive full-history replay hits token ceilings. 4. Conversation Summaries: compress older turns into rolling summaries. 5. Response Streaming: deliver tokens through SSE so users see progress immediately.

Stages 6–9 — platformizing the service

6. Multi-User Scale: stateless chat servers behind a load balancer. 7. Authentication: every conversation belongs to a user or workspace. 8. Rate Limiting: Redis protects both cost and availability. 9. LLM Gateway: central routing, retries, provider abstraction, and failover replace direct provider calls from the app tier.

Stages 10–14 — grounding the model in external knowledge

10. File Upload Support: assets land in object storage. 11. Document Processing Pipeline: background workers parse, normalize, and chunk content. 12. RAG: retrieve relevant knowledge at query time. 13. Vector Database: specialized ANN search makes retrieval fast. 14. Prompt Construction: assemble system prompt + summary + retrieved chunks + recent turns within a strict token budget.

Stages 15–20 — becoming a production assistant

15. Memory: keep durable facts across sessions. 16. Tool Calling: models request structured functions. 17. Agent Workflows: loop across multiple tool steps. 18. Cost Optimization: semantic cache, complexity-based routing, and token enforcement. 19. Reliability & Failover: circuit breakers, retries, provider failover, graceful degradation. 20. Production-Scale Architecture: multi-region deployment, global routing, replicated data services, and operational observability.

What forces each later stage

Each stage exists because the previous one breaks in a specific way. Persisting conversation history invites retries and refreshes — that's what forces turn identity and idempotency in. Streaming invites disconnects mid-generation — that's what forces an explicit assistant-message lifecycle and partial-response recovery. Adding RAG invites "which chunk, whose permission, indexed when" questions the moment retrieval is wrong — that's what forces provenance, authorization, and index lifecycle. Adding tools invites duplicate side effects the moment a retry or a repeated model call happens — that's what forces durable tool execution state and idempotency. Adding agents invites long-running, crash-prone workflows — that's what forces checkpointing, budget propagation, and resumability. And before multi-region, the system needs an explicit conversation write-ownership model — otherwise "deploy globally" quietly becomes "which region's write wins" the first time two regions see the same conversation.

Conversation quality depends on giving the model the right context, in the right order, within the model's context window. This is where many AI products quietly fail.

Recent turns are necessary but not sufficient

The last few turns preserve conversational continuity, but older turns still matter when they establish goals, constraints, or prior conclusions. The answer is not 'store everything forever in the prompt'; it is a layered context strategy.

Rolling conversation summaries

After every N turns or when the prompt crosses a threshold, a background summarizer compresses old turns into a structured summary: goals, decisions, entities, unresolved questions, and tool results. The next prompt includes that summary plus only the recent turns. This keeps context semantically rich while bounding token growth.

Context window budgeting

A practical budget might allocate 10% to the system prompt and policies, 15% to rolling summaries and memory, 35% to retrieved chunks, 30% to recent turns, and leave 10% reserved for tool results or unexpected growth. If a section exceeds budget, truncate or rerank that section first rather than blindly chopping the end of the whole prompt.

Context layers and trust

Distinguish trusted system/policy instructions, product/workspace instructions, the conversation summary, memory, retrieved external content, recent messages, tool results, and the current request as separate layers — physical ordering in the prompt is not by itself a security boundary between them. Retrieved content and tool results are untrusted data that happens to be useful evidence, not instructions the model should treat with the same authority as the system prompt (full mechanics in Context Construction, Architecture Deep Dive).

Summary lifecycle

A summary covers a known, recorded message range. Summary generation should read a stable range of already-committed turns rather than racing with turns still arriving concurrently, and a failed or stale summary must not corrupt conversation history — the raw turns remain durable and the summary can always be rebuilt from them.

Concurrent turns

What happens if the same conversation receives two user turns concurrently from two tabs or devices before either assistant response completes? Pick and state a policy explicitly — serialize the second turn behind the first, branch both into separate assistant responses, or reject/queue the second turn — rather than silently constructing both prompts from the same stale conversation snapshot and appending both responses nondeterministically afterward (see Concurrent Conversation Turns in Architecture Deep Dive).

Streaming is not cosmetic. It meaningfully changes perceived latency, cancellation behavior, and failure handling — and it is delivery, not durability.

Why SSE fits this product

Assistant output is fundamentally server-push text over a single request. SSE gives simple one-way streaming, HTTP infrastructure compatibility, ordered event framing, and reconnect support — a simpler operational model than a full duplex socket for most chat use cases. Cancellation and new user messages still travel as ordinary HTTP requests; SSE only needs to carry the server→client direction.

Partial persistence, not durability-by-delivery

SSE delivering a token to the client does not mean that token is durably stored. A practical design: create the assistant row as generating, buffer deltas, persist periodic checkpoints (not necessarily one write per token), commit the final response as completed, and mark disconnect/cancel/failure with the appropriate terminal or partial state. The persistence cadence is a trade-off between write load and recovery fidelity (see Persist Every Token vs Periodic Checkpoint vs Final Only in Trade-offs & Decisions).

Disconnect and resume

A client disconnect does not automatically mean the provider stopped generating — cancellation should propagate, but the server has to handle the race where output was already in flight. If exact-stream resume is supported, it needs event IDs and a retained event buffer plus Last-Event-ID; if it isn't, the client instead fetches the persisted (possibly partial) assistant state and reconnects or retries from there. State whichever behavior the design actually implements rather than assuming resume works implicitly.

Retry after partial output

If the provider fails after streaming part of an answer, do not automatically route the rest to a second provider and concatenate the output — that risks presenting two incompatible generations as one seamless answer. Fail the current attempt visibly, restart as a new attempt, or use an explicit provider-aware continuation strategy if the product specifically supports one (see Partial Generation Recovery in Architecture Deep Dive).

Moderation and buffering

Some teams stream raw provider deltas immediately. Others hold a short safety buffer so moderation can inspect content before releasing it. This is a product trade-off between fastest first token and strongest real-time controls (see Stream Immediately vs Safety/Quality Buffer in Trade-offs & Decisions).

File support should never block the chat path on heavy parsing. Upload is synchronous; understanding is asynchronous.

Upload path

The frontend first asks the Chat Service for a signed upload target, then writes the file directly to S3. That keeps large blobs off the synchronous app path and lets object storage absorb spikes naturally.

Document processing pipeline

An object-created event or upload-complete message places work on an async queue. Workers detect type, virus-scan if required, parse content, normalize encoding, chunk text, and persist chunk metadata. Failed jobs retry with backoff; poison files move to a dead-letter queue for inspection.

User-visible status

Uploads move through an explicit state machine — UPLOADED → SCANNING → PARSING → CHUNKING → EMBEDDING → INDEXING → READY, with failure/retry states representable at each stage rather than one generic failed bucket (full mechanics in Document Ingestion State Machine, Architecture Deep Dive). The conversation UI should surface this explicitly so users understand why retrieval is not available yet.

Idempotent ingestion

Object-storage events and queue messages can be delivered more than once. Workers key on upload ID, a content/version hash, and a pipeline generation so a redelivered event confirms a completed stage rather than creating duplicate chunk or index generations.

Replacement and deletion

A re-upload creates a new version rather than mutating the old one. Deleting the source should remove or tombstone old chunks/vectors, invalidate retrieval and cache references pointing at them, and preserve audit history only where policy requires it — a deletion that leaves the old content reachable through retrieval or cache is not really a deletion.

RAG converts external knowledge into a retrieval problem. The assistant stops pretending to remember every document and instead fetches only the pieces that matter for the current turn.

Embedding generation

Each chunk goes through an Embedding Service that produces a dense vector. Chunk size is a trade-off: larger chunks preserve context but reduce retrieval precision; smaller chunks improve matching but may lose coherence. A range like 300–800 tokens with overlap is a reasonable illustrative starting point, not a universal best practice — the right size depends on content structure, the embedding model, query style, answer granularity, and citation needs.

Query-time retrieval, hybrid and query-aware

On every eligible user turn, the system may rewrite the query for retrieval (resolving pronouns and conversational context into a standalone query), embed it, search the vector database within the correct namespace, and — where dense search alone misses exact identifiers, error codes, or rare terms — combine it with lexical/BM25 matching and metadata filters before reranking (see Hybrid Retrieval in Architecture Deep Dive). Retrieval can be skipped for small-talk turns to save money and latency.

Authorization and provenance travel with retrieval

A chunk is only eligible if the current requester can still access the underlying resource right now — namespace membership alone is not a complete security model (see Retrieval Authorization in Architecture Deep Dive). Every retrieved chunk should carry enough metadata to answer which source, which version, where in the source, when indexed, what permissions applied, and what score led to its selection — this is what makes citations trustworthy rather than decorative (see RAG Provenance & Citations).

Vector search infrastructure is a workload decision

Approximate nearest-neighbor indexes such as HNSW keep semantic lookup fast even when the corpus is large. The honest comparison, though, is indexed vector search in a relational database versus dedicated vector infrastructure — not a dedicated ANN index versus an unindexed linear scan. A relational database with an appropriate vector extension/index can be operationally acceptable at meaningful scale depending on workload (see Postgres + Vector Index vs Dedicated Vector DB in Trade-offs & Decisions).

Index consistency and evaluation

Metadata and vectors are usually two separate systems that must stay reconciled — a document deleted from metadata but still searchable as a vector is a real, common failure (see Vector/Metadata Index Consistency). And retrieval quality should be measured on its own terms — recall/relevance, reranker quality, context precision, groundedness, and citation correctness are five different signals, not one "RAG quality" number (see RAG Evaluation).

Long-term memory is different from conversation history. History records what happened in one thread. Memory stores durable facts that should influence future threads.

What belongs in memory

Preferences, enduring identity facts, recurring project context, and stable constraints belong in memory. Ephemeral requests do not. 'Use TypeScript examples' is memory. 'Explain the last upload from five minutes ago' is conversation context.

Extraction and the write lifecycle

After selected turns, a lightweight classifier or model decides whether durable facts were expressed. That extraction produces a candidate, not an automatic durable fact — a candidate becomes validated/accepted, then active only after confidence scoring and category constraints (and, where the product offers them, user controls) confirm it. Memory is derived data, not a verified oracle; treat model-extracted statements accordingly rather than automatically durable-storing them.

Retrieval

On new requests, prompt construction queries for relevant facts by user/workspace and injects only a handful of active, topically related memories — treat any specific count as an illustrative starting point rather than a fixed rule.

Conflict handling

Memories can become stale or contradictory. Prefer an explicit newer user correction over an older inferred memory, prefer an authoritative profile source where one exists, and deactivate the superseded entry rather than leaving contradictory memories equally active for prompt construction to pick between arbitrarily. Support versioning, confidence, source attribution, and deactivation rather than treating memory as an append-only truth oracle.

Deletion propagates

Deleting a memory has to affect the memory store, any derived embeddings/indexes, relevant caches, and future prompt construction — not just the primary record (see Data Deletion Does Not Reach Derived Stores in Failure & Scale Scenarios).

Tool calling turns the assistant from a text generator into a task performer. Agent orchestration adds repeated decision-making on top of that primitive — and both need explicit safety boundaries the model cannot talk its way around.

Structured tool calls and the tool contract

The model should never emit arbitrary code for direct execution. Instead it selects from a catalog of declared tools, each with a name/version, a typed argument schema, a result schema, an auth scope, a side-effect class, a timeout, a retry policy, idempotency support, a maximum output size, and a confirmation policy. The tool-execution service validates arguments against the schema, authorizes access against the platform's own record of what this user/session can do — never against what the model's arguments claim — executes in a sandbox matched to the requested capability, and returns bounded, structured results.

Idempotency and ambiguous outcomes

The model may request the same tool more than once, and the platform itself may retry after a timeout — either path must not execute a payment, email, or ticket-creation call twice. Give every invocation a stable ID, persist execution state, and pass a downstream idempotency key where the external system supports one (see Tool-Call Idempotency in Architecture Deep Dive). Sometimes the outcome is genuinely unknown — the external system may have already acted before a response was lost to a timeout. Don't blindly retry a non-idempotent action in that case; use a status lookup or reconciliation, or surface an explicit "outcome unknown" state (see Ambiguous Tool Outcomes).

Confirmation boundary and result trust

For actions requiring confirmation: the model proposes, the platform presents the normalized action, the user approves, and execution uses the approved parameters — the model cannot mutate a confirmed action afterward. Tool results, meanwhile, are data the model reasons over, not privileged instructions — a hostile string inside a tool result deserves exactly as much suspicion as the same string inside a retrieved document (see Tool Output Trust Boundary).

Agent Orchestrator as a durable state machine

Simple chat uses at most one tool step. Harder tasks — search several systems, inspect results, then write code or summarize findings — need an Agent Orchestrator that tracks explicit state: current goal, step number, the model's decision, the tool invocation and result, a checkpoint, budget consumed, and an eventual stop reason — not just an informal "plan → act → observe" narrative. Not every agent needs to be durable, but long-running or side-effect-heavy workflows should be able to survive a worker restart, a provider timeout, or a user reconnect by resuming from the last committed checkpoint rather than repeating side effects that already happened.

Budget propagation

A parent request should carry budgets for tokens, model calls, tool calls, wall-clock time, and monetary cost — and nested or delegated sub-agent work must consume from that same bounded budget rather than resetting its own limits, or a single ambiguous request can recursively spawn far more work than any sane ceiling allows.

A production AI assistant is one of the few mainstream systems where architecture decisions can change cost by an order of magnitude without changing traffic volume.

Semantic cache, scoped deliberately

Traditional caches match exact keys. Chat requests rarely repeat exactly, so the system can use a semantic cache: embed the new query, find a similar prior question plus a matching context fingerprint (scope, model/policy version, corpus version, personalization state), and reuse the answer when similarity and freshness thresholds are high enough. This is not a blanket reuse policy — good candidates are stable public factual transformations and deterministic low-personalization tasks; personalized, permission-sensitive, rapidly-changing, or action-oriented requests should bypass the cache by default (see Semantic Cache Safety in Architecture Deep Dive).

Cost attribution

Track cost by request/turn, user/workspace, model, retrieval, embeddings, reranking, tools, and agent workflow — a single blended "cost per message" figure hides which lever actually moves the number.

Model routing

Not every request deserves the most expensive model. Short factual Q&A, formatting, and routing can use the cheaper/faster model. Ambiguous reasoning, code-heavy tasks, or multi-step planning can route to stronger or specialized models. The LLM Gateway makes this routing a platform capability instead of scattered app logic.

Token budget enforcement

Set hard prompt ceilings per route. If retrieval returns too much or the conversation is too long, trim or summarize before the provider call. The platform should reject or reshape an oversize prompt rather than discovering the problem after paying for a failed request.

Cost vs quality

Do not automatically down-route to a cheaper model solely because budget was exceeded — quality and policy requirements are constraints, not knobs to trade away silently. Reasonable responses to budget pressure include rejecting the request, asking the user to narrow scope, truncating safely, choosing a cheaper compatible model, or disabling optional work — chosen deliberately, not defaulted to the cheapest option that technically completes.

Reliability for AI assistants means more than 'the API is up.' It means the user still gets a useful experience during provider errors, retrieval failures, and regional incidents — and that retries never turn a transient failure into a duplicate side effect.

Retry classification

Retry only when it's semantically safe, and the failure class determines that: a connection failure before the provider accepted the request is usually safe to retry; a timeout before any output is usually safe; a failure after partial output was already shown is not a blind-retry case (see Partial Generation Recovery); a failure after a tool call was emitted needs tool-call idempotency, not a provider retry; quota/rate-limit errors need backoff or rerouting, not immediate retry; deterministic invalid-request and context-length errors need the request reshaped, not repeated unchanged; provider safety/policy rejections need a product decision, not an automatic loop.

Hedged requests, qualified

Hedging (racing a duplicate request and taking whichever returns first) can reduce tail latency for eligible read-like inference calls, but it doubles cost while both requests run, complicates cancellation, can produce inconsistent outputs between the two racing attempts, and is unsafe around tool execution or other side effects unless carefully scoped to prevent both attempts from acting. It is not a generic provider-resilience mechanism to apply broadly — treat it as a narrow, deliberately scoped optimization.

Provider failover

The LLM Gateway wraps every provider with timeouts, retry-class-aware policy, circuit breakers, and health-based routing. Failover to an alternative must check capability compatibility first — required modality, tool schema support, context size, output format, policy constraints, latency/cost, and model quality — since a fallback model that can't support the tools or context the original request needed will fail differently, not more gracefully. Failover may degrade behavior, and that degradation should be observable, not silent.

Admission control

When inference capacity saturates, scaling out stateless APIs alone does not create more provider inference capacity. Use bounded queues, concurrency limits, per-tier/tenant fairness, deadlines, load shedding, backpressure, and provider-quota awareness so the system degrades predictably instead of letting queue depth grow until everything times out together.

Graceful degradation

If the Vector Database is down, answer from recent turns and summary — and make clear that the answer may not include document context, rather than silently answering as if grounding were unaffected. If memory retrieval fails, ignore memory and continue. If tool execution fails, return a partial answer that explicitly says the action could not complete. The system should degrade feature-by-feature, not fail all-at-once.

Multi-region production

At global scale, deploy chat APIs and gateways in multiple regions behind geo-routing. Keep conversation writes home-region-local, replicate durable state asynchronously across regions, and fail over to a secondary region for degraded operation — without implying that asynchronous replication provides zero-RPO failover. Object storage and vector namespaces need their own multi-region recovery plans, independent of the conversation store's.

By the final stage, every component exists because a simpler version broke first. The production architecture is broad, but it is still explainable because each box solved a named problem.

Four logical paths

Turn path: Client → Chat API → persist/idempotently establish turn → Context Orchestrator → optional Retrieval + Memory → LLM Gateway → Provider → normalized stream → persisted assistant message → Client. Knowledge path: Upload → Object Storage → Queue → Parser → Chunker → Embedding → versioned retrieval index. Action path: model tool request → tool policy/auth → durable invocation state → tool executor → external system → result/reconciliation → model. Agent path: turn → agent state machine → model/tool steps → checkpoints → bounded budgets → final result. A control/quality plane runs alongside all four — rate/admission control, safety/policy, evaluation, tracing, cost accounting, and routing configuration — rather than being bolted onto any single path.

Component map