ChatGPT / AI Assistant — Architecture Deep Dive

Everything from Send to a terminal state is one logical turn. Naming its stages explicitly is what makes retries, cancellation, and partial failure reasonable to build.

The stages

1. Request identity is established (conversationId, userMessageId, turnId, requestId, idempotencyKey). 2. The user message is durably persisted. 3. An assistant message placeholder/attempt is created (assistantMessageId). 4. Context is built (summary, memory, retrieval, recent turns). 5. The provider executes, optionally invoking tools. 6. Output streams to the client. 7. The assistant message is finalized and persisted with usage/cost metadata. 8. The turn reaches a terminal status.

Terminal states are not optional

A turn should end in an explicit state — completed, failed, cancelled, or partial — rather than an implicit absence of further events. This is what lets a client, a retry, or an operator later answer "what actually happened to this turn?" without guessing from log fragments.

A browser retry after a timeout must not silently create a second logical turn.

The mechanism

The client sends a stable idempotencyKey with POST /conversations/{conversationId}/messages. If the server has already accepted that key, it returns the existing turn's identity (and current status/stream location) instead of creating a new user message and a new assistant attempt. This applies even if the original request's response was lost due to a network failure — the server-side record, not the client's belief about what happened, is authoritative.

Where this matters most

Refreshing a tab mid-stream, a mobile client retrying after a dropped connection, and a proxy or load balancer retrying a request it couldn't confirm was delivered are all ordinary events, not edge cases. Without idempotency, each of these can fork a conversation into duplicate user turns and duplicate assistant generations — including duplicate tool workflows if a retry lands after a tool was already invoked.

Two devices open on the same conversation can both send a message before either assistant response completes. The architecture needs an explicit policy, not accidental behavior.

Policy options

Serialize — the second turn waits until the first reaches a terminal state, then builds context from the now-current conversation state. Branch — both turns proceed from the same parent point and create separate assistant-response branches the user can compare or pick between. Reject/queue — the platform explicitly tells the second device a generation is already in progress and offers to queue or cancel-and-replace it.

What to avoid

The unsafe default is silently building both prompts from the same stale conversation snapshot and appending both assistant responses nondeterministically afterward — that produces a conversation whose message order doesn't reflect what either client actually saw when it sent its turn. Pick one policy explicitly and make ordering deterministic (see Trade-offs & Decisions for the recommended baseline).

Prompt construction is the control plane of answer quality — and, once retrieval and tools are involved, the control plane of trust.

Context layers and trust classes

A robust assembly distinguishes: (1) trusted system/policy instructions, (2) product/workspace instructions, (3) conversation summary, (4) memory, (5) retrieved external content, (6) recent messages, (7) tool results, (8) the current request. Physical ordering in the prompt is not, by itself, a security boundary — retrieved content and tool results are data, not instructions, and that distinction needs to be preserved through how the model is instructed to treat each section, not just where each section sits.

Provenance travels with the block

Each injected block should retain metadata identifying its source, trust class, timestamp/version, applicable permissions, and — for retrieved content — its relevance score. This is what makes context debugging possible: knowing why a chunk or memory was included, not just that it was.

Budget allocation and overflow

Each section gets an illustrative token budget — treat any specific percentage split as a starting assumption, not a universal rule. If a section exceeds its budget, trim or rerank that section specifically (drop the lowest-relevance retrieved chunk, shorten the oldest summary detail) rather than running one global truncation pass that can delete the newest or most important content by accident.

Conversation storage looks simple — ordered turns keyed by conversation — but ordering, lifecycle, and idempotency all live here.

Deterministic ordering

Do not rely solely on created_at for deterministic ordering — clock skew and concurrent writers can produce ties or out-of-order timestamps. Use a monotonic per-conversation sequence number alongside a stable message ID, and track branch/parent relationships explicitly where regeneration or concurrent turns create more than one lineage.

Message and turn lifecycle

Persist the user turn first, allocate the assistant message ID as a placeholder, then stream provider deltas against that message, tracking an explicit status (pending, generating, waiting_for_tool, completed, failed, cancelled, partial). This pattern is what makes partial streams, retries, and duplicate-send handling auditable after the fact.

Retention and archival

Hot recent history stays in the conversation store for low-latency reads; older raw turns can be archived to cheaper storage while summaries remain hot. The summary's covered-range boundary (see Conversation Summary Consistency) is what makes it safe to archive the turns underneath it.

A summary is derived data with a precise boundary — it has to say exactly which messages it covers, or prompt construction can silently double-count or skip turns.

Covered range

A summary should record the message sequence (or turn boundary) it covers through — analogous to a snapshot's version boundary. The summarization job reads a stable range of already-committed turns; it must not race with turns still arriving concurrently, or it can summarize a state that was never actually the conversation's committed state at any single point in time.

Failure and rebuild

A failed or stale summary job must not corrupt conversation history — the raw turns remain the durable source of truth, and a summary can always be regenerated from them. If a summary omits or distorts an important constraint (see Summary Corruption / Stale Summary in Failure & Scale Scenarios), the fix is to rebuild it from the durable range it claims to cover, not to patch the summary text directly.

Streaming is a protocol and recovery problem as much as a model problem. Delivery and durability are different guarantees.

SSE is delivery, not durability

SSE delivers incremental events to a connected client. It does not, by itself, establish whether the user turn was persisted, whether assistant output was durably stored, whether a tool action happened, or whether the request can be safely retried — those guarantees belong to the turn's application state (see AI Turn Lifecycle), not to the transport.

Event sequence and identity

Every event should carry the stable turn/message identity and, where reconnect is supported, an event sequence or ID for client-side reconciliation. Each event is framed as event: + data: lines; the server can emit message_started, content_delta, tool_call_started, tool_call_result, status, usage, error, and done over the same ordered stream.

Backpressure and cancellation

A slow-reading client needs bounded server-side buffers; past that threshold, terminate the stream rather than buffering unbounded output in memory. Client disconnect should propagate as a best-effort cancellation of the downstream provider call, but the server must handle the race where the provider was already producing output when the cancel arrived.

A provider can fail after 300 tokens were already shown to the user. What happens next is a product decision with real correctness implications.

The safe default

Preserve the partial output that was actually streamed, mark the attempt partial or interrupted rather than silently discarding it, and let the user retry or regenerate explicitly. Do not automatically route the remainder of the answer to a second provider and concatenate it onto the first provider's partial output — two different models mid-sentence is not a seamless answer, it's an unlabeled Frankenstein response.

Continuation as an explicit capability

If a product wants true continuation (resuming generation rather than restarting), that requires an explicit, provider-aware continuation strategy — not an implicit assumption that any fallback model can pick up where another left off. Absent that explicit support, treat a partial-then-failed generation as an interrupted attempt requiring a fresh regenerate.

Ingestion needs named, versioned states so retries and reprocessing are safe rather than accidental.

The stage sequence

UPLOADED → SCANNING → PARSING → CHUNKING → EMBEDDING → INDEXING → READY, with explicit failure/retry states representable at each stage rather than one generic "processing failed" bucket.

Idempotent stage transitions

Object-storage events and queue messages can be delivered more than once. Workers key on upload ID, a content/version hash, and a pipeline generation/version so a redelivered event re-confirms a completed stage transition instead of creating duplicate chunk or index generations. This is the same idempotency discipline as tool-call execution, applied to background ingestion.

Replacement and deletion

Re-upload creates a new version rather than mutating the old one in place. Deleting the source should tombstone or remove old chunks/vectors, invalidate retrieval and cache references pointing at them, and preserve audit history only where policy actually requires it (see Vector Index Consistency and Deleted Document Still Appears in Retrieval).

Embeddings convert text into vectors that semantic search can operate on — and embedding-model identity is part of the data, not an implementation detail.

Chunking is illustrative, not a rule

A range like 300–800 tokens is a reasonable starting assumption, not a universal best practice. The right chunk size depends on content structure, the embedding model, typical query style, desired answer granularity, and citation needs — a legal contract and a chat transcript do not chunk the same way.

Batching and model versioning

Embedding calls are typically cheaper and faster when batched by model and namespace. Every embedding should record which embedding model/version produced it — this is what makes an eventual model migration (see Embedding Model Migration) a controlled, versioned operation instead of a silent corruption.

Partial batch failure and rate limits

A batch embedding call can partially fail. Track success per chunk, not per batch, and retry only the failed subset idempotently against the ingestion state machine's stage tracking rather than re-running the whole batch.

Vector search is where retrieval becomes interactive — the system needs high recall without linear scans over millions of chunks.

ANN basics

Approximate nearest-neighbor search uses index structures such as HNSW to navigate a graph of vectors instead of comparing against every stored embedding, turning an otherwise expensive semantic search into a low-latency online operation.

Relational vector indexing is a legitimate option

A relational database with an appropriate vector extension/index can be operationally acceptable at meaningful scale, depending on workload — the honest comparison is indexed vector search in relational storage versus dedicated vector infrastructure, not a dedicated ANN index versus an unindexed linear scan. Which one to choose is a workload question (query volume, corpus size, operational simplicity), not a rule that retrieval-heavy systems must always graduate off a relational store.

Namespaces and filters

Production vector search rarely queries one giant global corpus — it filters by user, workspace, document set, permission scope, language, or recency so retrieval stays relevant and access-safe. Permission filtering belongs here structurally, but must still be revalidated at query time (see Retrieval Authorization) because namespace membership alone can go stale.

Dense semantic search is not the only retrieval mechanism a production system should have — and for some query shapes, it is the wrong one.

What dense search misses

Exact identifiers, error codes, part numbers, names, and rare terms often embed poorly relative to how distinctly they matter to the user's query — a dense-only system can miss the one chunk that contains the exact error code the user pasted, even though it's semantically "close enough" to plenty of irrelevant chunks.

Combining signals

A production retrieval system may combine lexical/BM25 matching, dense ANN search, metadata filters, and reranking — using dense search for semantic recall and lexical search for exact-term precision, then letting reranking resolve the combined candidate set. This is not a mandatory architecture for every small product; it is a response to a specific recall gap that dense-only retrieval has.

The user's literal message is often a poor retrieval query on its own, especially deep into a conversation.

Techniques

Retrieval may use the original user query as-is, a standalone rewritten query that resolves pronouns and implicit context from the conversation ("what about the second one" → a self-contained question), or multi-query expansion that issues several related queries and merges results. Filters derived from conversation state (a document the user already referenced, a time range they mentioned) can narrow the search further.

The risk

Query rewriting can change intent — a rewrite that over-generalizes or mis-resolves a reference can retrieve confidently wrong evidence instead of no evidence at all, which is a worse failure mode than an empty retrieval result. Query rewriting should be observable and evaluated on its own (see RAG Evaluation), not assumed to be a strict improvement.

Tenant namespaces are useful, but they are not the entire security model for retrieval.

Query-time enforcement

A chunk must only be eligible for retrieval if the current requester can still access the underlying resource right now — not merely at the time the document was indexed. A namespace filter narrows the search space; it does not, by itself, guarantee the requester's access hasn't since been revoked.

Propagating permission changes

Permission changes and deletions must propagate to the retrieval layer safely — either by checking current authorization at query time against a permission record, or by designing the index itself to be permission-safe (partitioned or filtered such that a revoked chunk cannot surface even under a stale cache). Whichever approach is chosen, cross-tenant or cross-user leakage through retrieval is a security incident, not an ordinary relevance bug (see Wrong Memory Injected and its retrieval analogue in Failure & Scale Scenarios).

A citation is not generated merely because the model happens to mention a source name — it needs an unbroken chain of metadata behind it.

The provenance chain

source → version → chunk/location → retrieval result → prompt evidence → answer citation. Each link needs to carry forward: which source document produced this chunk, which version of that document, where within it, when it was indexed, what permissions applied to it, and what retrieval/reranking score or rank led to its selection.

What this enables

This metadata is what makes citations trustworthy in the UI, makes debugging a wrong answer tractable ("which chunk led the model here, and was it actually relevant?"), supports evaluation of retrieval quality against ground truth, and makes deletion and re-indexing safe — you can find every citation that depended on a chunk before you remove it.

Relational metadata and the vector index are usually two separate systems — keeping them consistent is a real distributed-systems problem, not a footnote.

Failure shapes

Relational metadata commits but the vector write fails (or vice versa); an old document version remains searchable after a new version was indexed; a deletion is applied to metadata but not propagated to the vector store; an embedding-model migration leaves old and new vectors in incompatible spaces inside the same index.

The fix pattern

Use an index generation/version tag on both the metadata row and the vector entry, make indexing operations idempotent (safe to retry without duplicating), run periodic reconciliation that compares metadata and vector-store state and flags or repairs drift, and support a rebuild-from-source capability for when reconciliation alone isn't enough. For embedding-model migrations specifically, run dual indexes during a controlled cutover rather than mixing embedding spaces silently in one index (see Embedding Model Migration in Failure & Scale Scenarios).

A retrieval system can be fully available and still be silently failing the product — RAG evaluation is what catches that.

Separate what you're measuring

Retrieval recall/relevance (did the right chunks come back at all), reranker quality (did the best candidates end up on top), context precision (how much of what was included in the prompt was actually useful), grounded-answer quality (did the model actually use the evidence correctly), and citation correctness (does the cited source actually support the claim) are five different signals, not one "RAG quality" number.

Ground truth beyond clicks

"Retrieval hit ratio" alone is a weak proxy — a chunk can be topically related without actually answering the question. Combine offline evaluation sets (curated question/expected-evidence pairs) with online signals (explicit feedback, groundedness checks, citation-click patterns), rather than relying on user follow-up rate alone as a reliability signal — follow-ups can mean the answer was wrong, or that the user simply had a related second question.

Memory adds personalization across sessions, but only if extraction, storage, and retrieval are disciplined — memory is derived data, not a truth oracle.

The write path

candidate → validated/accepted → active → superseded/deactivated/deleted. A model-extracted statement becomes a candidate, not an automatic durable fact — product rules, a confidence threshold, category constraints, and (where applicable) explicit user controls decide whether it graduates to active memory.

Retrieval

Memories are retrieved by scope (user/workspace), subject, topical relevance, recency, active state, and sensitivity/product policy. Inject only a handful of active, relevant memories — treat any specific count as an illustrative starting point, not a universal rule.

Deletion propagates

Deleting a memory must remove it from the memory store, any derived embeddings/indexes built on it, relevant caches, and future prompt construction — a deletion that only touches the primary record but leaves it reachable through a cache or a semantic-cache-derived answer isn't really a deletion (see Data Deletion Does Not Reach Derived Stores).

Memories can become stale or contradictory — the system needs a deterministic way to prefer the current truth over an outdated one.

Deterministic preference rules

Prefer an explicit newer user correction over an older inferred memory; prefer an authoritative profile source over an extracted inference where one exists; deactivate the superseded entry rather than leaving contradictory memories equally active and letting prompt construction pick one at random. If the user says "I no longer use Python," the old preference memory should be deactivated, not left to compete with the new statement on equal footing.

Provenance supports correction

Every memory should retain source attribution (which conversation/turn produced it, or which system) and a supersedes/conflict relationship to any memory it replaces. This is what makes "why does the assistant think that about me" answerable, and what makes correcting a wrong extraction a targeted fix instead of a full memory wipe.

Tool calling is a protocol between model and platform. The platform decides what is allowed; the model only proposes.

Tool contract

Each tool declares a name/version, an argument schema, a result schema, an auth scope, a side-effect class (read-only, reversible, irreversible), a timeout, a retry policy, idempotency support, a maximum output size, and a confirmation policy. The model may still hallucinate fields or invalid types, so the executor must reject invalid payloads deterministically against the declared schema.

Authorization belongs to the platform

The model cannot grant itself permissions by emitting arguments that look authorized — every tool invocation is authorized against the current user/session's actual capabilities by the platform, independent of what the model's arguments claim.

Structured, bounded results

Tool results return as structured messages with size limits, not unbounded prose — an unbounded tool result is a hidden context-window bug just as much as unbounded conversation history is.

A payment, email, or ticket-creation call must not execute twice merely because a network response was lost — and the model itself may request the same tool more than once.

The mechanism

Create a stable invocation ID per tool call. Persist execution state before or around execution as appropriate for the side-effect class. Pass a downstream idempotency key to the external system where it supports one. If a duplicate invocation arrives (same invocation ID, or the model re-emits an equivalent call), return the already-persisted result instead of executing again.

Why the platform retry path matters here specifically

The platform itself may retry a tool call after a timeout, independent of the model deciding to call the tool again — both paths need the same invocation-ID-based dedupe, or a platform-level retry becomes exactly the duplicate-side-effect bug this section exists to prevent (see Duplicate Tool Execution in Failure & Scale Scenarios).

Sometimes the orchestrator genuinely cannot tell whether a side effect happened.

The scenario

1. The assistant asks an email service to send a message. 2. The service sends it. 3. The network times out before the response reaches the orchestrator. 4. The orchestrator cannot tell whether the send succeeded or failed.

The safe response

Do not blindly retry a non-idempotent action on ambiguous timeout — that risks sending twice. Instead: use the downstream idempotency key if the external system supports one (a retry becomes safe even on real ambiguity), perform an operation-status lookup against the external system if one is available, and fall back to an explicit "outcome unknown" state surfaced to the user or an operator if neither option resolves the ambiguity. "Outcome unknown" is a legitimate terminal state — better than a confident wrong guess in either direction.

For actions that matter, the model proposes and the user (or policy) decides — and the platform, not the model, enforces that boundary.

The confirmation flow

1. The model proposes an action with specific parameters. 2. The platform presents the normalized action to the user for approval — not the model's raw text, but the structured parameters that will actually execute. 3. The user approves. 4. Execution uses the approved, immutable parameters, or requires re-confirmation if anything material changed between proposal and execution.

What must not happen

The model must not be able to mutate a confirmed action's parameters after approval — approval is a contract on specific parameters, not a general license for the model to act. If the model wants to change something after approval, that is a new proposal requiring new confirmation.

Tool output can contain hostile or manipulated content — a scraped webpage, a hostile email body, a poisoned search result — and it must not gain system-instruction authority just because it enters the prompt.

The rule

Tool results are data, exactly like retrieved document chunks (see the retrieved-content trust boundary), and should be framed to the model as evidence to reason about, not as instructions to follow. Text inside a tool result that says "ignore previous instructions" is exactly as untrustworthy as the same text inside a retrieved PDF.

"Plan → act → observe" is a useful mental model but not a durable design on its own — a real agent needs explicit, inspectable state.

What the state machine tracks

Current goal, step number, the model's decision at this step, the tool invocation (if any) and its result, a checkpoint of progress so far, budget consumed against the allotted budget, and — when the loop ends — an explicit stop reason (success, error, budget exhausted, loop detected, user cancelled).

Not every agent needs to be durable — but long-running or side-effect-heavy workflows need to survive a crash without repeating work that already happened.

What has to survive

A worker restart, a provider timeout mid-step, a user reconnecting after closing the tab, or a delayed tool response should all be recoverable from the last committed checkpoint — not force the whole workflow to restart from step one, which risks re-triggering side effects (like re-sending an email) that already completed.

Two classes of agent

Interactive, ephemeral agents (a quick multi-tool lookup inside one chat turn) can reasonably live in-process without durable checkpointing. Long-running or side-effect-heavy workflows need durable checkpoints specifically so resume-after-crash never replays a completed side effect (see Agent Worker Crash in Failure & Scale Scenarios, and Ephemeral vs Durable Agent in Trade-offs & Decisions).

A parent request should carry a bounded budget that nested work consumes from — not a budget every sub-agent resets for itself.

What gets budgeted

Tokens, model calls, tool calls, wall-clock time, and monetary cost. A top-level agent request establishes the ceiling for all of these; any sub-agent or nested delegated work draws down from the same bounded budget rather than starting with its own fresh allotment.

"Same tool, similar arguments" catches the obvious runaway loop, but real agents fail in subtler repeating patterns too.

What to detect beyond the obvious

Repeated near-identical tool calls, yes — but also repeated conversation/goal state with no measurable progress between steps, cyclic delegation (agent A delegates to B, which delegates back to A), repeated failed actions with no adaptation, and straightforward budget exhaustion (steps, tokens, time, or cost limit reached) as its own stop condition, not just a fallback after other detectors miss it.

Stopping safely

When any of these trigger, stop with an explanatory fallback and inspectable state — the step-by-step record from Agent State Machine is what makes it possible to later diagnose why the loop happened, not just that it was stopped.

The LLM Gateway is the anti-corruption layer between product code and a constantly changing provider ecosystem.

Responsibilities

Provider normalization (different APIs, streaming formats, tool-call protocols, and error surfaces unified into one internal contract), auth/credential management, routing, retry policy, quota enforcement, circuit breaking, observability, and usage accounting.

Retrying an LLM call is not one uniform behavior — the right response depends entirely on what already happened before the failure.

Failure classes and their retry behavior

Connection failure before the provider accepted the request — safe to retry. Timeout before any output was produced — usually safe to retry. Failure after partial output was already streamed to the user — do not blindly retry and splice (see Partial Generation Recovery); surface an interrupted attempt instead. Failure after a tool call was emitted — retry must go through tool-call idempotency, not a blind provider retry, since a tool may have already executed. Quota/rate limit — back off and possibly reroute, not an immediate retry. Deterministic invalid-request errors — never retry unchanged. Context-length rejection — requires shrinking the prompt, not retrying the same one. Provider safety/policy rejection — should not be retried as-is; it needs a product decision, not an automatic loop.

Routing is a quality policy as much as a cost policy — "cheapest model likely to succeed" is an incomplete rule on its own.

What routing has to weigh

Capability, modality, tool support, context length, latency target, quality requirement for this task class, safety/policy constraints, current provider health, quota headroom, and cost — in that rough order of hard constraints before cost becomes the tiebreaker among compatible options.

Capability classes over named models

Prefer describing routing targets by capability class — a fast/general model, a high-capability reasoning model, a coding-specialized model, a multimodal model, a fallback provider — rather than a timeless named tier like "GPT-4-class," which ages out as the market shifts. Named examples are fine when clearly labeled as illustrative and dated.

A router can save 30% on cost while silently degrading answer quality — cost metrics alone will never show you that.

What to track per route

Route selected, task class, an outcome/evaluation score for that task class, latency, cost, whether a fallback was triggered, and user retry/escalation rate for that route specifically — not just in aggregate.

Semantic similarity between two questions is not sufficient justification for reusing an answer — the cache decision needs a much richer fingerprint.

What belongs in the fingerprint

User/workspace scope, model/policy version, system-prompt version, retrieved-corpus/version, tool availability, freshness window, personalization/memory state, safety classification, locale, and task type. Two semantically similar questions from different scopes, different corpus versions, or different personalization state are not safe to answer identically.

Bypass classes

Highly personalized, permission-sensitive, action-oriented, or rapidly-changing-information requests should bypass answer caching by default rather than relying on the fingerprint to always catch every edge case — some request classes are simply too risky to cache regardless of similarity score, because a wrong reuse there is a privacy or safety incident, not just a slightly-off answer (see Semantic Cache Contamination in Failure & Scale Scenarios).

When inference capacity saturates, scaling out stateless APIs does not create more provider inference capacity — a different control is needed.

Controls

Bounded request queues rather than unlimited waiting work, explicit concurrency limits per provider/route, per-tier and per-tenant fairness so one workspace can't starve everyone else, deadlines that let the system reject or defer a request whose response would arrive too late to be useful, load shedding for the lowest-priority traffic first, backpressure signaled back to callers, and provider-quota awareness feeding directly into admission decisions rather than discovering quota exhaustion only via failed calls.

A request can return HTTP 200 and still fail the product — quality has to be observable, not just uptime.

Three categories

Tracing across the whole turn

Use distributed tracing across turn → retrieval → model → tool → model → stream so a single slow or wrong answer can be diagnosed as a single connected trace rather than reconstructed from five separate logs after the fact.

Stateless APIs are easy to run globally; conversation write ownership and durable state are the harder part.

The baseline

Stateless APIs and the LLM Gateway deploy in multiple regions behind geo-routing. Conversation writes have a home region/write-locality preference per conversation. Durable data (conversation store, retrieval index, memory) replicates asynchronously across regions for disaster recovery and read availability, not necessarily as an active-active write target.

Failover semantics

On a regional failure, in-flight streams in the failed region are lost and need to be resumed or retried from persisted state, not silently continued. Reconnecting clients reroute to a healthy region and rebuild from the last durably persisted turn. Retrieval indexes need their own replication/availability plan independent of the conversation store's.