ChatGPT / AI Assistant — Interview Perspective

Ask directly. Start from plain chat regardless of the answer, then add whichever planes the interviewer confirms are in scope, narrating each addition as a response to a new requirement.

• If tools are in scope, do actions need user confirmation before executing? — Pulls in the confirmation-boundary discussion early.
• Is cross-session memory required, or is per-conversation context enough? — Determines whether Memory Architecture belongs in the design at all.

If yes, RAG needs a provenance chain from source to chunk to prompt to citation, not just retrieval — mention this early so it doesn't get bolted on later.

• What happens if the model mentions a source name without an actual retrieval hit behind it? — Tests whether the candidate treats citation as a generated claim or a verified provenance link.

If yes, distinguish interactive ephemeral agents from durable, checkpointed workflows early — this changes the state-machine design, not just the UI.

• What durability guarantee do agent actions need if a worker crashes mid-task? — See Agent Checkpointing & Resume in Architecture Deep Dive.

State the invariant up front: a submitted user turn must be durably persisted before any model work begins, and a completed tool action must never be silently lost or duplicated.

Cost and provider capacity scale with tokens processed and generated, not with request count alone — a single request can cost 50x another depending on context size and output length.

• How would you estimate daily inference cost without naming a specific vendor price? — Expect a parametric formula: input_tokens × input_price + output_tokens × output_price, labeled illustrative.

Roughly request rate × average generation duration — this bounds provider concurrency needs and streaming-connection count, independent of total daily volume.

One user action can trigger many nested model calls and tool invocations; without a budget, an agent-heavy feature can multiply inference volume well beyond what per-user request counts suggest.

• How do you bound that amplification? — See Agent Budget Enforcement in Architecture Deep Dive.

Inference throughput and per-account quota are the provider's capacity to manage; your infrastructure owns admission control, queuing, and routing around that external ceiling — autoscaling your API tier cannot create more provider capacity.

A client-generated idempotency key on the message submission lets the server recognize the retry and return the existing turn instead of creating a duplicate.

• What if the retry arrives after a tool call already executed? — Idempotency must be enforced at the turn level, not just the HTTP layer.

So the user's turn is never lost even if the model call fails — persistence-before-work is a core reliability invariant, not an optimization.

Pick one explicit policy — serialize, branch, or reject/queue — and apply it deterministically rather than letting database write order decide silently.

• Which policy would you pick for a consumer chat product vs. a collaborative workspace? — Tests whether the candidate treats this as a product decision, not just a technical one.

Messages form a tree via parent_message_id; regenerate creates a sibling branch rather than mutating history, so prior attempts remain inspectable.

A monotonic sequence plus a stable message ID and parent/branch reference — created_at timestamps alone are not sufficient under concurrent writes or clock skew.

Token cost and latency scale with prompt size, and context windows are finite — full replay stops scaling long before most conversations get long, which is why summaries and selective retrieval exist.

A summary covers a known, already-committed range and does not race with new turns; a failed summarization job must not corrupt the underlying history, since summaries are rebuildable derived state.

• What if the summary loses an important constraint the user stated earlier? — See Summary Corruption / Stale Summary in Failure & Scale.

Context is built from explicit layers with a defined priority and trust class — system/policy instructions first, then product instructions, then summary, memory, retrieved content, recent turns, tool results, and the current request — trimmed by budget, not by physical order alone.

Provenance metadata per context item — source, trust class, timestamp/version, permissions, relevance score — makes this inspectable instead of a black box.

SSE gives simple one-way streaming, works over standard HTTP infrastructure, has built-in event framing, and supports reconnection — not because browsers 'automatically support' it as some inherent advantage. WebSocket earns its place for bidirectional realtime needs like voice.

Cancellation propagates best-effort to the provider, but the server must handle the race where the provider already produced output — persist what was generated and mark the turn's terminal state explicitly rather than leaving it ambiguous.

Not a database write per token — buffer deltas in memory and persist via periodic checkpoints, landing on a terminal state (completed, partial, failed, cancelled) rather than writing every token.

• What are the trade-offs of per-token writes vs. periodic checkpoints vs. final-only? — See Stream Persistence Strategy in Trade-offs & Decisions.

Only if the protocol retains event IDs and a buffer, or the client sends Last-Event-ID and the server supports it — otherwise the client falls back to fetching the persisted assistant-message state.

Do not silently splice a second provider's continuation onto the first's output. Mark the attempt interrupted/partial and expose an explicit retry or regenerate action.

Ingestion: upload → scan → parse → chunk → embed → index, each stage tracked and retryable. Query time: embed the query, retrieve candidates, optionally rerank and filter by metadata/permissions, then construct the prompt.

Vector search captures semantic similarity beyond exact keyword match, but it misses exact identifiers, error codes, names, and rare terms — which is exactly why hybrid retrieval combining lexical and dense search exists.

• When does lexical search beat embeddings outright? — Exact codes, IDs, rare tokens, and precise phrase matches.

Initial ANN retrieval optimizes for recall over a large candidate set; reranking applies a more expensive, more accurate relevance model to a small candidate set to improve precision before it reaches the prompt.

Authorization is enforced at query time — a chunk is only eligible if the current requester can still access the underlying resource right now, not because it once belonged to an accessible namespace.

Through a provenance chain: source document/version → chunk/location → retrieval result → prompt evidence → answer citation. A citation isn't valid merely because the model mentions a source name.

Deletion must propagate to metadata, the vector index, and any cache entries referencing it — not just the source record, or the deleted content keeps surfacing in retrieval.

Old and new vectors are not comparable in the same space — build a separate versioned index, backfill, validate quality, then cut over; never mix vectors from two models in one index.

Separate metrics for retrieval relevance/recall, reranker quality, context precision, grounded-answer quality, and citation correctness — a single 'retrieval hit ratio' hides which stage is actually failing.

History is the raw record of this conversation; RAG retrieves external documents; memory is a small set of durable, cross-session facts about the user extracted and curated over time — distinct storage, distinct lifecycle.

Not every extracted statement — apply product rules, confidence thresholds, provenance, category constraints, and user controls before a candidate becomes an active memory.

Deterministic conflict rules — prefer explicit newer user corrections and authoritative profile sources, deactivate the superseded memory rather than keeping both equally active.

Deletion must reach the memory store, any derived embeddings/indexes, caches that might have memoized a memory-influenced answer, and future prompt construction — not just one row.

That's a security incident, not ordinary degraded service — it requires halting the affected path and auditing exposure, not just a bug-fix ticket.

The model proposes a structured call; the platform validates arguments, checks authorization, and executes — the model never holds credentials or grants itself permission.

Against a versioned schema defined in the tool contract, before execution — untrusted or malformed arguments are rejected the same way any API input would be.

A stable invocation ID, persisted execution state, and a downstream idempotency key passed to the external system where supported — a duplicate invocation returns the persisted result instead of re-executing.

• What if the external system doesn't support idempotency keys? — Fall back to operation-status lookup and reconciliation.

Don't blindly retry a non-idempotent action — use status lookup or reconciliation, and fall back to an explicit 'outcome unknown' state if neither resolves it.

By risk/action class — read-only actions can run automatically; irreversible or consequential actions require the platform to present a normalized action for explicit user approval, and execution must use the approved, immutable parameters.

No — tool results may contain hostile content and should not automatically gain system-instruction authority, the same trust-boundary problem as retrieved documents.

An agent iterates: model decision → tool invocation → observation → next decision, potentially many steps, requiring its own durable state machine rather than a single request/response.

Detect repeated state, repeated tool calls, no-progress patterns, and cyclic delegation — backed by hard step/token/time/cost limits as a final backstop, stopping with an inspectable state.

The parent request carries a budget for tokens, model calls, tool calls, wall-clock time, and cost; nested work consumes from that same bounded budget rather than resetting limits for each sub-agent.

Durable checkpointing of goal, step number, decisions, and results lets a workflow resume on a different worker without redoing completed side effects.

Tool-call idempotency makes any accidental re-execution safe even if the resume logic has a bug — checkpointing and idempotency are complementary, not substitutes for each other.

It centralizes provider abstraction, routing, retries, failover, observability, and policy enforcement — justified as routing/policy/observability needs grow, not solely by organization size.

• When is a gateway unnecessary? — A small product may reasonably call one provider through a thin adapter.

Before any output or side effect, a retry is generally safe; after tokens have been shown or a tool invoked, the same retry can duplicate visible output or a side effect — retry policy must be aware of what already happened.

Check compatibility first — required modality, tool schema support, context length, output format, policy constraints — then weigh latency, cost, and quality; failover that silently drops capability is a worse failure than the outage.

Route-specific evaluation and task-success metrics per task class — a router can look cheaper in aggregate while quietly failing harder tasks routed to an insufficient model.

Quota headroom should feed into routing and admission decisions proactively — quota exhaustion is a distinct failure mode from a provider outage and needs its own detection and response.

Prompts vary turn to turn with conversation-specific content, so exact-match hit rates are low outside narrow repeated-query cases — semantic caching targets a broader class of near-duplicate queries.

Reusing a cached answer for a semantically similar (not identical) query — dangerous whenever the request is personalized, permission-sensitive, action-oriented, or safety-sensitive, since similarity alone doesn't guarantee the cached answer is still correct or authorized for this requester.

User/workspace scope, model/policy version, system-prompt version, corpus/version, tool availability, freshness requirement, personalization/memory state, safety classification, locale, and task type — a fingerprint missing any of these can cause contamination.

A cached answer tied to permissions or a data version that has since changed must not be served — treat any ambiguity about scope match as a cache miss, not a hit.

Yes for general chat — retrieval is optional in prompt construction — but the response path must know retrieval failed and expose degraded grounding rather than silently answering a document-specific question without the document.

Only requests compatible with the fallback's capabilities — modality, tool support, context size, output format — everything else needs a different degraded path or explicit failure.

No — don't start new model work that can't attach to durably persisted turn state, and don't casually substitute an in-memory store like Redis as the source of truth for conversation writes.

Requests whose deadline can't realistically be met, and lowest-priority/lowest-tier traffic — bounded queues with explicit deadlines beat unlimited waiting for everyone.

Make each plane (retrieval, memory, tools, agents) independently optional in the request path, with its own failure signal, so one subsystem's outage narrows functionality instead of taking down chat entirely.

Treat retrieved content as untrusted evidence, never as instructions — maintain the instruction/data boundary in how context is framed, and keep tool authorization independent of anything claimed in document text.

No — a tool result is data to reason about, not a new instruction source; execution trust (the platform ran the tool correctly) is separate from content trust (what the tool returned).

At the platform/tool-execution layer and at retrieval query time — never as something the model can claim or grant to itself through prompt text.

Enforce scope filtering at the query layer with defense in depth, re-check permissions at retrieval time rather than trusting stale namespace membership, and include cross-tenant isolation tests in the standard suite.

A deletion workflow must reach conversation rows, object storage, chunk metadata, vector indexes, memory, semantic caches, derived summaries, and logs per retention policy — not just the primary record.

Usually not by default — a home-region write-authority model for conversation state, with globally deployed stateless edge/API components, is simpler to reason about than active-active; active-active adds real conflict-resolution cost that must be justified.

It's lost — the client reconnects to a healthy region and falls back to fetching persisted turn state rather than resuming the exact stream.

Non-zero — some recent writes can be lost in a regional failure. Never claim zero-RPO failover unless the design actually pays for synchronous cross-region durability.

In the same durable, replicated store as conversation state — otherwise a long-running agent workflow can't resume correctly after failover.

Via their own replication or rebuild-from-source path, tracked with the same index-generation mechanism used for reconciliation — retrieval availability during a regional event is a separate concern from conversation-store failover.

The idempotency key on the retried request matches the existing turn — return its current state instead of creating a duplicate.

Mark the attempt partial/interrupted, keep the partial text visible, and offer explicit regenerate — don't auto-splice a second model's continuation.

Don't resend — check status via the downstream idempotency key or an operation-status lookup, and surface 'outcome unknown' if neither resolves it.

The second invocation is recognized by its stable ID/idempotency key and returns the first's persisted result instead of executing again.

Tombstone the chunk at the metadata layer immediately so retrieval authorization filters it out, then clean up the vector asynchronously and reconcile.

That's a versioned migration, not a hot-swap — the old index stays authoritative until a new, separately built index is validated and cut over.

Raw history is still durable and separate from the summary — rebuild the summary from the recorded covered range using corrected logic.

Treat it as a contamination incident — invalidate the entry, tighten the fingerprint dimension that was missing, and audit exposure.

Check route-specific task-success evaluation, not cost dashboards — roll back the routing change for the affected task class until evaluation passes.

Nested work must consume from the same propagated budget as the parent — if the budget is exhausted, the recursion stops regardless of how 'productive' it looks.

Route around it to compatible alternate capacity, apply priority-based admission for high-value traffic, and treat this as distinct from a provider outage.

The agent resumes from its last committed checkpoint in the replicated durable store, and tool-call idempotency ensures the already-completed side effect isn't repeated.