ChatGPT / AI Assistant — Trade-offs & Decisions

The assistant needs incremental token delivery, status events, and cancellation, but the client does not necessarily need full duplex real-time messaging.

Option A — Server-Sent Events

Simple HTTP semantics, ordered server-to-client events, native browser support, straightforward load-balancer behavior, and a natural fit for "POST a message, then stream the answer." Cancellation and new user messages still travel as ordinary HTTP requests alongside the SSE channel — SSE only needs to carry the server→client direction.

Option B — WebSocket

Full duplex communication is more flexible for collaborative editing, voice, or bidirectional live control, but it adds connection lifecycle complexity, infrastructure tuning, and a broader protocol surface than a single chat turn needs.

Decision

The trade-off is that SSE is less general. If the roadmap includes synchronized cursors, live co-agents, or voice streaming, WebSocket may eventually become necessary for those specific surfaces. For a text-first assistant, SSE is the right default, and other HTTP-streaming alternatives can also serve narrower needs without moving the whole product to a socket model.

The system must ground answers in large external corpora and long conversations. This is not actually an either/or choice — long context and RAG solve overlapping but distinct problems.

Not a false binary

Use recent conversation turns directly, use retrieval for large external corpora that don't fit (or don't deserve to occupy) a full context window, and use summaries for older conversational state. Some tasks genuinely benefit from full-document context when the document fits the window and exact global reasoning across the whole document is required — RAG's chunk-based retrieval can actually hurt a task like "does this contract contain any clause about X anywhere" if the right clause gets chunked away from its surrounding context.

Option A — Long context only

Operationally simple: no embedding pipeline, no vector database, no retrieval tuning. But prompt size and cost rise with corpus size, and irrelevant history or document text often harms answer quality even when it technically fits.

Option B — RAG

Adds ingestion complexity but keeps prompts focused, cheaper, and easier to fit inside deterministic token budgets. It scales to document sets that dwarf any single context window, at the cost of retrieval-tuning risk.

Decision

The system needs both durable metadata and semantic search. The real comparison is indexed vector search in relational storage versus dedicated vector infrastructure — not indexed search versus an unindexed scan.

Option A — Postgres with a vector extension/index

Operationally convenient: one backup story, one metadata home, fewer moving parts, and — with an appropriate vector index — genuinely acceptable ANN performance for many workloads, not just tiny ones.

Option B — Dedicated Vector DB

Better ANN performance ceiling, richer namespace/index tuning, and more predictable low-latency retrieval at very large scale or very high query concurrency. Purpose-built systems handle the largest embedding sets more gracefully.

Decision

The trade-off either way is operational: a dedicated vector store adds a system and a consistency-coordination problem between metadata rows and vector entries (see Vector/Metadata Index Consistency in Architecture Deep Dive); staying in Postgres longer avoids that but caps how far retrieval scale can grow before a migration becomes necessary.

Should every request use the same flagship model, or should the platform route by task complexity and capability needs?

Option A — Single model

Simple mental model, consistent quality behavior, and straightforward evaluation. But it often overpays for easy tasks, gives no resilience when that one provider or model class degrades, and locks the whole product to one vendor's safety and capability behavior.

Option B — Multi-model routing

Cheaper for simple tasks, more resilient under provider incidents, and able to exploit specialized coding or reasoning models where they excel — at the cost of routing policy, capability-compatibility checks, and ongoing evaluation discipline (see Routing Quality Evaluation in Architecture Deep Dive).

Decision

The trade-off is evaluation surface area, quality-consistency risk across model classes, provider lock-in avoidance versus multi-provider integration overhead, and the need to track safety-behavior differences between models routed to the same task.

Application code can call providers directly, or it can depend on a central LLM Gateway that standardizes routing, retries, and policy.

Option A — Direct provider access

Fast to prototype, fewer hops, and reasonable for one team shipping one feature with one provider. A small product can legitimately call one provider through a thin adapter without needing a full gateway. But provider logic spreads across the codebase as more teams and features are added, and each one reimplements errors, rate limits, usage accounting, and failover independently.

Option B — Central gateway

Adds one internal hop but centralizes normalization, auth, policy, retries, circuit breakers, observability, and cost controls.

Decision

The trade-off is that the gateway becomes critical infrastructure and must itself be highly available — but that is usually preferable to duplicating fragile provider logic across every team once more than one depends on LLM access.

Long conversations need continuity, but sending the full transcript every turn quickly becomes too expensive and noisy.

Option A — Full history in every prompt

Simple and intuitively safe, but token costs rise monotonically and context windows fill with stale details that no longer help the current question.

Option B — Rolling summaries + recent turns

Requires summarization logic and lifecycle management (see Conversation Summary Consistency), but keeps prompt size bounded while preserving important semantics from older turns.

Option C — Selective retrieval from conversation history

For very long conversations, targeted retrieval of specific older turns (treating conversation history itself as a small retrieval corpus) can surface a precise old detail that a summary might have compressed away, complementing recent turns and the rolling summary rather than replacing either.

Decision

The trade-off is summary quality and, if added, retrieval-tuning effort. Bad summaries can erase important nuance, so summary generation and refresh timing deserve real attention regardless of whether selective retrieval is layered on top.

Chat inputs rarely repeat byte-for-byte, so exact-key caches have limited value for answer reuse — but semantic reuse needs real constraints, not just a similarity score.

Option A — Traditional exact cache

Cheap, predictable, and easy to invalidate, but only catches identical prompts or near-identical normalized strings — that misses most real conversational repetition.

Option B — Semantic cache

Embeds queries and matches semantically similar requests, improving hit rates materially, but needs thresholds, freshness rules, and a full context fingerprint (scope, versions, personalization) to stay safe — semantic similarity alone is not sufficient justification for reuse (see Semantic Cache Safety in Architecture Deep Dive).

Decision

The trade-off is evaluation risk: exact caches are deterministic, while semantic caches need careful tuning and explicit bypass classes to avoid reusing an answer that is merely similar, not correct for this context (personalized, permission-sensitive, or action-oriented requests especially).

When a user uploads a file, should the chat request wait for parsing and indexing, or should ingestion happen in the background?

Option A — Synchronous ingestion

Simple UX because the document is immediately ready if it succeeds, but request latency becomes hostage to parser speed, OCR complexity, and embedding throughput for anything nontrivial.

Option B — Asynchronous ingestion

Fast upload acknowledgment, scalable workers, retries, and isolation from the chat path — at the cost of explicit status tracking so the user knows when retrieval is ready.

Decision

The trade-off is eventual readiness for the async path. Users may ask about a document before indexing completes, so the product needs clear processing states and graceful messaging regardless of which path a given upload took.

Should raw provider output stream to the user the instant it arrives, or should the platform hold a short buffer to inspect content first?

The options

Fastest time-to-first-token — stream provider deltas essentially unmodified. Small moderation/validation buffer — hold a short window (a sentence, a few hundred milliseconds) so lightweight moderation or structural validation can run before release. Full-response validation before release — wait for the complete answer, validate it, then release — strongest control, weakest perceived latency.

Decision

When a provider call fails, should the gateway retry the same provider or fail over to a different one?

What the decision actually depends on

The failure class (see Provider Retry Semantics in Architecture Deep Dive) — a transient timeout before any output is a different situation than a failure after a tool call already executed. Whether output was already emitted to the user. Capability compatibility of the failover target (tools, context length, modality, structured-output support). Relative cost of the failover target. And whether any tool/side-effect state is already in flight, which failover cannot safely ignore.

Decision

Should retrieval rely on dense semantic search alone, or combine it with lexical matching and metadata filters?

The comparison

Dense retrieval alone is simpler to operate and covers most conversational, semantically-phrased queries well. Hybrid retrieval (dense + lexical/BM25 + metadata filters + reranking) recovers exact-term precision — identifiers, error codes, names, rare terms — that dense-only search can miss, at the cost of running and maintaining an additional retrieval path.

Decision

How often should streamed assistant output actually be written to durable storage during generation?

The options

Persist every token — strongest recovery fidelity, but high write amplification for what is often a low-value granularity. Periodic checkpoint — buffer deltas and persist every N tokens or M milliseconds, giving strong recovery with bounded write load. Final only — simplest, but a crash mid-stream loses the entire partial output, and partial-stream recovery (see Partial Generation Recovery) becomes weak or impossible.

Decision

Not every agentic task needs the same durability guarantees — the right orchestration model depends on how long the task runs and how much it touches the outside world.

The options

An interactive, short agent loop (a few tool calls inside one chat turn) can reasonably use simpler in-process or state-store orchestration without durable checkpointing. A long-running or side-effect-heavy workflow (multi-step research, actions with real external consequences) needs durable checkpoints and resumability, so a crash doesn't force a restart that risks repeating a completed side effect.

Decision

Should the model be allowed to execute a proposed tool call automatically, or does the action require explicit user approval first?

The decision axis

Risk and reversibility of the action class. Read-only actions (search, lookup, read a file) can generally run automatically since there's nothing to undo if the model chose poorly. Irreversible or consequential actions (send an email, create a financial transaction, delete data) should require explicit confirmation, because a wrong tool call there has a real-world cost that automatic execution can't take back.

Decision

Should each conversation have one home region that owns its writes, or should multiple regions accept writes for the same conversation?

The options

Home-region writes — each conversation has a write-authority region; stateless API and gateway components deploy globally and route conversation writes back to that region. Simpler ordering and failover reasoning; users far from the home region incur extra latency on writes. Multi-region active writes — multiple regions can accept writes for the same conversation concurrently, reducing local write latency but requiring real cross-region coordination to keep message ordering and turn state consistent.

Decision

Maximizing cache hit rate is not the objective — correctness, permission isolation, and freshness bound how much can safely be cached.

The tension

Every additional request class made cache-eligible saves cost, but also expands the surface for a stale or cross-scope answer reaching the wrong user (see Semantic Cache Contamination in Failure & Scale Scenarios). Good candidates for caching are stable public factual transformations, deterministic low-personalization tasks, and expensive retrieval results under a safe, versioned scope. Poor candidates are personalized/memory-dependent responses, permission-sensitive private data, rapidly changing information, tool/action requests, and safety-sensitive decisions.

Decision