AI Product Design Patterns

Watch six recurring design patterns emerge from very different products — ChatGPT, GitHub Copilot, Netflix, enterprise search, and voice assistants — then see how they combine into one repeatable AI product architecture.

Choose the AI application architecture pattern from the job: fixed-schema extraction needs deterministic code with a model at the fuzzy boundary, questions about your documents need retrieval-grounded answers, edits a person reviews need a copilot, choosing among existing items needs candidate retrieval and ranking, changes to external systems need a tool-using workflow or bounded agent, and slow media needs an asynchronous generation pipeline. Pick the AI pattern by looking at the job the product must do and what can go wrong, not at the model you would like to use. Five properties of the job decide almost everything: how uncertain the input is, what type of data goes in and comes out, whether the system causes side effects in other systems, how much control the user must keep, and how much latency the user will tolerate. A fixed record to fill, a question about your documents, an ordering of existing items, a draft a person will edit, a multi-step change to an external system and a ten-minute video are different jobs, and each has a small, well-understood architecture that fits it. The best design is the smallest pattern that meets the requirement, and anything beyond it has to earn its place with measured errors.

This lesson is the synthesis of the other twelve in the applications track. It does not re-teach them. It shows how to choose between them, how the six core patterns look when you strip them to their skeleton, which cross-cutting mechanisms every pattern needs, and how to compose patterns without letting any one component exceed its authority. Where a chapter owns the details, such as AI search, recommendation or business agents, we link to it and spend the words on the decision. For the general definition of an AI application and its components, start with AI Applications Overview; this lesson owns the architecture choice and trade-offs.

A finance team receives about 20,000 supplier invoices per month. The process is fixed: read the document, extract vendor, invoice number, line items, tax and total, check that the arithmetic holds, match against a purchase order, then either post to the ledger or send to a clerk. An enthusiastic team builds an “autonomous invoice agent.” A model is given tools (read_pdf, lookup_po, post_to_ledger, email_clerk) and a goal: “process this invoice correctly.” The demo works. In production it becomes hard to operate, and the reasons are structural rather than a matter of model quality.

First, the sequence was never uncertain. Every invoice goes through the same steps in the same order, so letting a model choose the next step spends tokens on a decision the team already knows. Second, the agent’s loop re-sends a growing history on every turn. Third, arithmetic is a rule, not a judgement. A model asked whether 12 × 48.50 + 3 × 120.00 + 25.00 plus tax equals the printed total will usually be right, but “usually” is the wrong guarantee for a ledger, when ten lines of code give a guarantee that is exact. Fourth, you cannot write a regression test for “the agent behaves,” because the trace differs between runs. Fifth, post_to_ledger is a side effect whose permission now depends on a model choosing to call it.

The deterministic-first design splits the job along the line where uncertainty really lives. Reading a messy scanned page into fields is genuinely fuzzy, so a model does that, once, and must return a fixed schema. Everything after extraction (arithmetic, currency and date validation, purchase-order matching, the posting decision) is ordinary code with unit tests. The model never sees a posting tool.

Here is a cost and latency comparison. Every number below is illustrative: assume a price of $3 per million input tokens and $15 per million output tokens, and use per-invoice token counts that a team might measure on its own traffic.

Agent design: 7 sequential model calls per invoice
  input  = 7 calls x 4,500 tokens (history grows)  = 31,500 tokens
  output = 7 calls x   250 tokens                  =  1,750 tokens
  cost   = 31,500 x $3/1M + 1,750 x $15/1M
         = $0.0945 + $0.02625 = $0.12075 per invoice
  month  = 20,000 x $0.12075 = $2,415
  latency ~ 7 x 2.5 s = 17.5 s (sequential, illustrative)

Deterministic-first design: 1 extraction call
  input  = 2,000 tokens, output = 300 tokens
  cost   = $0.006 + $0.0045 = $0.0105 per invoice
  month  = 20,000 x $0.0105 = $210
  latency ~ 3 s model + a few ms of rule code

ratio of cost: 0.12075 / 0.0105 = 11.5x

The saving is a side benefit. The larger gain is that the safe version has a bounded set of failure modes you can enumerate: extraction returned a wrong field, a rule rejected a correct invoice, or the purchase-order system was unavailable. Each of these has a test and a metric. The agent version adds one more category, the model deciding to do something unexpected, and it is the one that you cannot enumerate.

Product conversations tend to start with technology (“we need RAG,” “let’s add an agent”). A more reliable approach is to classify by what the user is trying to get done and then read the pattern off. Nine jobs cover most of the products in this track. Classify or extract turns unstructured input into a label or record. Retrieve and answer answers questions from a body of documents. Generate and edit produces a draft that someone refines. Recommend or rank orders existing items for a user. A conversational assistant sustains a multi-turn dialogue. A copilot sits inside a human workflow and proposes. A tool-using workflow calls systems along a known path. A bounded agent chooses its own path inside strict limits. And a human-approved action adds a deliberate gate before any consequential side effect.

These are not mutually exclusive categories. The last four describe how much autonomy a system has, and the first five describe what it does with data. A real product usually combines one from each group, which is why the composition section comes later. What separates them are five axes:

• Uncertainty — Is the input ambiguous (handwriting, free text) or the path ambiguous? Ambiguous input needs a model; ambiguous path needs planning; neither needs code.
• Data type — Fixed-schema records, passages of text, behavioural events about items, or media. This decides between extraction, retrieval, ranking and generation.
• Side effects — Read-only output can be wrong and corrected. A refund, an email or a deploy cannot always be undone. Side effects decide how much code, permission and approval surrounds the model.
• User control — Whether the user is the decision-maker (copilot), a supervisor (approval gate) or a bystander (automation). This is a product and legal choice, not a model property.
• Latency — Sub-second interaction, a few seconds of waiting with progress, or minutes of background work. Latency decides synchronous versus queued execution and how many model calls fit.

The tree below applies these axes in order of increasing machinery. It deliberately asks the cheapest question first. Notice that the generative patterns appear late; if an earlier question answers “yes,” you stop there.

Two caveats keep the tree honest. It selects the primary pattern, and a product usually needs one or two supporting ones: a handbook assistant is retrieval at its core, but the chat interface that wraps it brings conversation state from the chatbots lesson. Also, a “yes” at a late question does not override an earlier “yes”: a feature that both fills a schema and changes state is an extraction pattern followed by a gated action, not an agent.

The pattern places the model on exactly one side of a narrow interface. Upstream of the interface sits the messy world (pixels, free text, inconsistent layouts); downstream sits the business, which has rules. The model converts the first into the second, and a validator, written in code, decides whether the result is trustworthy enough to cross. The document-specific mechanics (OCR, layout, tables, review queues) live in the Document Intelligence lesson; this section shows the structural idea that transfers to any extraction problem.

A worked trace: invoice INV-2291

The model is asked for a strict JSON object and nothing else. Notice that it is asked for what is printed, including the printed total, not for the total it computes. Extraction and verification must be independent, or the check proves nothing.

{
  "invoice_number": "INV-2291",
  "vendor_name": "Northwind Packaging GmbH",
  "currency": "EUR",
  "lines": [
    { "description": "Shipping cartons 40x30", "qty": 12, "unit_price": 4850, "line_total": 58200 },
    { "description": "Pallet wrap 500mm",      "qty": 3,  "unit_price": 12000, "line_total": 36000 },
    { "description": "Freight",                "qty": 1,  "unit_price": 2500, "line_total": 2500 }
  ],
  "subtotal": 96700,
  "tax_rate": "0.075",
  "tax": 7253,
  "printed_total": 103953,
  "field_confidence": { "printed_total": 0.62, "tax": 0.91 }
}

The validator layer then runs. Lines: 12 × 4,850 = 58,200; 3 × 12,000 = 36,000; 1 × 2,500 = 2,500; sum = 96,700, which matches the printed subtotal. Tax: 96,700 × 0.075 = 7,252.5 cents. Here a rule must be chosen, and it belongs in code, not in a prompt: this vendor rounds half up, giving 7,253, so the extracted tax is accepted. Policy might instead accept a tolerance of ±1 cent because vendors round differently. The total is 96,700 + 7,253 = 103,953, which equals the printed total, and the invoice passes.

Now suppose the OCR read the final digit group badly and the model returned a printed total of 108,953. The arithmetic check fails by 5,000 cents (50.00 EUR) and the invoice is routed to a clerk with the failing rule, the expected value and the extracted value displayed. The model did not need to be “smart enough” to avoid the error, because the system was built so that its error is caught where it matters.

interface Line { qty: number; unit_price: number; line_total: number }
interface Extracted {
  lines: Line[]; subtotal: number; tax_rate: string; tax: number; printed_total: number;
}

type Check = { rule: string; ok: boolean; expected?: number; actual?: number };

// Tax rule belongs to code and policy: round half up to whole cents, allow 1 cent of vendor rounding.
function roundHalfUp(numerator: number, denominator: number): number {
  return Math.floor((2 * numerator + denominator) / (2 * denominator));
}

export function validate(inv: Extracted): Check[] {
  const checks: Check[] = [];
  inv.lines.forEach((l, i) => {
    const expected = l.qty * l.unit_price;
    checks.push({ rule: 'line_' + i, ok: expected === l.line_total, expected, actual: l.line_total });
  });
  const sum = inv.lines.reduce((s, l) => s + l.line_total, 0);
  checks.push({ rule: 'subtotal', ok: sum === inv.subtotal, expected: sum, actual: inv.subtotal });

  const ratePermille = Math.round(parseFloat(inv.tax_rate) * 1000); // 0.075 -> 75
  const tax = roundHalfUp(inv.subtotal * ratePermille, 1000);
  checks.push({ rule: 'tax', ok: Math.abs(tax - inv.tax) <= 1, expected: tax, actual: inv.tax });

  const total = inv.subtotal + inv.tax;
  checks.push({ rule: 'total', ok: total === inv.printed_total, expected: total, actual: inv.printed_total });
  return checks;
}

// route: every check must pass; otherwise a human sees the failing rules.
export const route = (cs: Check[]) => (cs.every((c) => c.ok) ? 'auto_post' : 'clerk_review');

Two design details deserve emphasis. The schema is enforced at the interface: invalid JSON or a missing field is a validation failure, not something downstream code guesses around (see Structured Outputs for how schema-constrained generation works). And model confidence is used as a routing hint, not proof: a low confidence (0.62 above) can send a passing invoice to review, while a high confidence never skips the arithmetic checks. Self-reported confidence is not a calibrated probability unless you have measured it against labelled data.

An employee handbook assistant answers questions such as “Can I carry over unused vacation days?” The model does not know your company’s policy, and a policy changes faster than a model is retrained, so the application retrieves the relevant passages at question time and gives them to the model as evidence. The model’s job is to write an answer from the evidence, not to remember the policy. For why this works, read Why RAG Exists; the stages are covered in the Retrieval Pipeline and Context Injection lessons, and the answer-engine variant in AI Search Systems.

One question, end to end

Maria, an employee in the Munich office, asks the question above. The system does not just embed the text. It first builds a retrieval request that carries her identity and attributes, because access control is enforced in the retrieval layer, not requested politely in the prompt. A policy that applies to US employees only must not even be a candidate.

request:
{ "query": "carry over unused vacation days",
  "filters": { "acl_groups": ["all-employees", "dept-engineering", "country-DE"],
               "status": "current", "effective_date_lte": "2026-10-28" },
  "top_k": 4 }

result (cosine similarity scores are illustrative):
[
 { "chunk": "HR-LEAVE-DE@v4#c07", "score": 0.83, "effective": "2026-01-01",
   "text": "Up to 5 unused days may be carried over until 31 March of the following year." },
 { "chunk": "HR-LEAVE-GLOBAL@v7#c02", "score": 0.79, "effective": "2025-04-01",
   "text": "Local policy takes precedence over this global summary." },
 { "chunk": "HR-LEAVE-US@v3#c05", "score": 0.0,  "excluded": "acl: country-US" }
]

The prompt then contains the question, the two passages with their stable chunk ids, and instructions to answer only from them and to cite ids. The model replies with an answer such as “Up to 5 unused days can be carried over until 31 March [HR-LEAVE-DE@v4#c07]” and a citation list. Two application steps follow. A verifier checks that each cited id was in the retrieved set and, ideally, that the quoted claim is supported by that passage. A citation that merely exists is not evidence that it says what the model claims; the verification of support is the harder step, discussed under citation faithfulness in the AI search chapter.

Abstention and freshness

If Maria asks “Can I sell back my vacation days?” and the best chunk scores 0.41, below a threshold your evaluation set established for this index, the correct output is “I can’t find this in the handbook” plus a route to HR. An answer written from weak evidence is exactly how a hallucination reaches a user. The threshold is a product setting you tune with labelled questions, because embedding scores from different models and indices are not comparable.

Freshness is an ingestion problem. Each chunk carries a document id, a version, a content hash, an effective date and a status. When HR publishes v5, ingestion marks v4 as superseded rather than leaving both searchable; when a document is deleted, its chunks are tombstoned and removed from the index, otherwise the assistant will quote a withdrawn policy with full confidence. Showing the effective date next to the citation lets a user see how current the answer is.

A copilot proposes and a person decides. The structure is chosen when the cost of an unreviewed error is high, when judgement and accountability must stay with a human, or when the model’s output is a draft whose quality is easier to judge than to produce. Contract review is a classic case: a lawyer is responsible for the outcome, but reading forty pages for deviations from the company’s standard positions is slow. The interaction design (what to show, how to keep users from rubber-stamping) is the subject of the Copilots lesson; the control mechanisms are in HITL & Guardrails. Here we look at the architecture that makes the human’s authority real rather than decorative.

The key artefact is the suggestion record. It is not a rewritten document; it is a structured proposal that names what it applies to, what it would change, and why.

{
  "suggestion_id": "sg_0042",
  "base_version": 1,
  "clause_id": "cl_14_liability",
  "span": { "start": 8120, "end": 8394 },
  "original": "Supplier's total liability is unlimited.",
  "proposed": "Supplier's total liability is limited to the fees paid in the preceding 12 months.",
  "reason": "Playbook rule LIAB-02 requires a liability cap.",
  "evidence": [{ "type": "playbook_rule", "id": "LIAB-02" }],
  "model_confidence": "medium",
  "status": "pending"
}

Three engineering decisions follow from the human being the decision-maker. First, evidence travels with the suggestion: the reviewer sees the rule or clause that triggered it, and can judge it in seconds instead of trusting an unexplained rewrite. Second, the system applies accepted edits by code, not by asking the model to regenerate the document. Regeneration can alter text nobody asked it to change, and the diff the lawyer approved would no longer be the diff that shipped. Third, edits are versioned and checked against the base.

function applySuggestion(doc: Doc, s: Suggestion, reviewer: User): Doc {
  if (s.base_version !== doc.version) throw new StaleSuggestion(s.suggestion_id); // text moved
  if (doc.text.slice(s.span.start, s.span.end) !== s.original) throw new StaleSuggestion(s.suggestion_id);
  const text = doc.text.slice(0, s.span.start) + s.proposed + doc.text.slice(s.span.end);
  audit.append({ suggestion: s.suggestion_id, by: reviewer.id, action: 'accepted', at: now() });
  return { ...doc, version: doc.version + 1, text };   // v1 stays immutable and restorable
}

The result is that the human remains responsible and in control, and the model’s contribution is measurable: the accept, edit and reject rates per rule tell you where it helps and where it is noise. A high accept rate with a very short time-to-accept deserves suspicion rather than celebration: it can mean reviewers have stopped reading, which makes the “human in the loop” a formality. Sampling accepted suggestions for a second review is a product control that checks the human step is real.

A short-video feed has a different job from every pattern above. There is no question to answer and no document to produce. The system must choose, for this user at this moment, about 20 items out of millions that already exist. Doing that with a generative model would be slow, expensive and unnecessary, because the output is an ordering of existing items, learned from behaviour. This is predictive ranking, not generative answering; the full treatment is in the Recommendation Systems lesson, and its retrieval primitives are in Embeddings and Similarity Search.

The funnel exists because scoring is expensive and the catalog is huge. Candidate generation uses cheap, high-recall sources: nearest neighbours of the user’s embedding, videos from followed creators, currently popular items in the user’s region, and recent uploads for freshness. Together they yield, say, 600 candidates (illustrative). The ranker, which can use richer features, estimates for each candidate something like the probability of meaningful watching and of a like. A common simple formulation is a weighted sum:

score = 0.7 x p_watch + 0.3 x p_like         (weights are a product decision)

video  topic    p_watch  p_like  score
A      cooking  0.85     0.75    0.7*0.85 + 0.3*0.75 = 0.82
B      cooking  0.80     0.80    0.7*0.80 + 0.3*0.80 = 0.80
C      travel   0.76     0.70    0.7*0.76 + 0.3*0.70 = 0.74
D      cooking  0.70     0.72    0.7*0.70 + 0.3*0.72 = 0.71   (scores rounded to 2 dp)

Sorting by score alone gives A, B, C, D: three cooking videos in the first slots, one of
them nearly a duplicate of A (similarity A-B = 0.90).

Greedy re-rank (MMR style): pick argmax of 0.7*score - 0.3*max_similarity_to_picked
  slot 1: A (nothing picked yet)                    -> A
  slot 2: B = 0.7*0.80 - 0.3*0.90 = 0.290
          C = 0.7*0.74 - 0.3*0.20 = 0.458
          D = 0.7*0.71 - 0.3*0.85 = 0.242           -> C (travel)
  slot 3: B = 0.290 ; D = 0.7*0.71 - 0.3*0.88 = 0.233  -> B
  slot 4: D
result: A, C, B, D

The distinction from generative answers is more than cosmetic. A feed item cannot be “hallucinated” in the sense of inventing a video, but the system can still fail in ways that are specific to ranking: feedback loops, where what is shown shapes what is clicked and so what is learned; popularity bias; cold start for new users and new items; and optimising a proxy (watch time) that diverges from what users actually value. Its evaluation is therefore offline ranking metrics plus controlled online experiments, not citation checks. If a product wants a generated natural-language explanation of a recommendation, that is a separate, optional generative step layered on top, and it must be grounded in the real ranking features, not invented after the fact.

When the product must read or change other systems, the model needs tools. Two designs share that ingredient and differ in who chooses the sequence. In a fixed workflow the developer writes the path (look up the order, check the policy, draft the reply) and calls a model at the steps that need language understanding. In a bounded agent the model chooses the next tool from an allowed set, inside limits that the runtime enforces. The agent earns its extra variance only when the right path genuinely depends on what each step discovers. Mechanics are in Tools & Tool Calling, The Agentic Loop and Production Agent Architecture; the business context is in Agents in Business.

Refund investigation: where an agent is justified

A customer writes that they were charged twice for order ORD-5531. Whether that is a duplicate charge, a legitimate pre-authorisation hold, or two separate orders depends on what the payment system shows, and what to do next depends on that finding. That fuzzy path is a reasonable agent job. The side effect, though, must be structured so the model can propose but never execute.

Four properties make this safe, and none of them is a prompt. Permissions: the runtime scopes every read to the customer on the case and exposes only read tools and propose_refund to the model; the actual issue_refund credential is never in the model’s reach. State: the case is a record with explicit states, so a crashed worker resumes from refund_approved rather than re-asking the model what to do. Approval: a rule in code (illustratively, any refund of $50 or more) holds the proposal until a human approves. Idempotency: the execution carries a key built from the case and charge, so a retry after a timeout returns the original refund instead of paying twice (see Idempotency).

// Exposed to the model:
propose_refund: {
  input:  { charge_id: string, amount_cents: integer, reason: string },
  effect: "creates a pending proposal; moves no money"
}

// Enforced by the runtime, regardless of what the model says:
assert(charge.customer_id === case.customer_id);          // authorisation
assert(amount_cents > 0 && amount_cents <= charge.amount_cents - charge.refunded_cents);
assert(!alreadyProposed(case.id, charge_id));              // duplicate proposal
if (amount_cents >= 5000) requireHumanApproval(proposal);  // threshold from policy config
idempotency_key = case.id + ':' + charge_id;               // e.g. CASE-7719:ch_8842
after execute: get_charge(charge_id).refunded_cents === amount_cents   // verified completion

Verification matters because an HTTP 200 from a payment call proves the request was accepted, not that the business outcome occurred. The runtime reads the charge back and only then moves the case to resolved; otherwise it raises the case to a human with the mismatch. The agent also has budgets (a maximum number of tool calls, a time limit, a spend limit) and a defined exit to a person when it is stuck, which turns an open-ended loop into a bounded one.

Contrast this with the invoice case from the start. If the refund process were “always look up charges, check duplicates, and propose a refund if duplicate,” a fixed workflow would be better, since it has the same tools and permissions but a developer-written path. The agent is justified here only because some cases need different follow-ups (a held authorisation, a split shipment, a chargeback already open) that cannot be listed in advance. Where you can list the branches, write them as code. Dependable infrastructure for this style of execution, such as queues, retries and compensation, is covered in the Saga pattern.

Generating a 30-second product video takes minutes, not milliseconds, and the result needs checking before anyone sees it. A request-response design (hold the HTTP connection open until the video is ready) fails for timeouts, retries and user experience alike. The pattern is a job: the request creates a record and returns immediately, workers run stages in the background, artifacts are stored as versions, and the client learns progress by polling or by events. Product-level differences between text, image, audio and video are covered in AI Content Generation; the queue and worker mechanics are in job processing style system design lessons.

A job record carries an id, an idempotency key from the client (so a double-click does not create two jobs and two bills), the inputs, the current state, and a manifest for every artifact version: prompt, model identifier, settings, seed where supported, and a content hash. That manifest is what makes a regeneration reproducible enough to debug and a published asset auditable. Regenerating creates version 2; version 1 stays, so a reviewer can compare and roll back.

Two details are easy to get wrong. Moderation is a pipeline stage with the authority to block, not a prompt instruction to “avoid unsafe content,” and it runs on the artifact itself (frames and audio), not just on the prompt that requested it. Cancellation is cooperative: the API records the request, and the worker checks it at stage boundaries or between chunks. Work already performed is not free, so the state machine should say what a cancelled job costs and ensure that a cancelled job never reaches publication even if a late worker finishes rendering. Rights, provenance and licensing questions about the content itself belong to the content chapter and to your legal review, not to the queue.

Beneath the six patterns sit ten mechanisms that recur. Each is simple to describe and expensive to retrofit, and each changes the design in a specific way. They are best understood as questions you ask of every model call in your system.

Context construction and grounding

Every model call receives a context that your code assembled: instructions, the user input, retrieved evidence, tool results and maybe history. Treat assembly as a function with a token budget, an ordering and a provenance for each piece, not as string concatenation. What the model can do is limited by what is in that window (see Context Window and System Prompts). Unbounded history is the usual failure: cost and latency rise with every turn while relevance falls. Design the context deliberately: summarise or drop old turns, keep durable facts in a store the application controls, and never treat conversational context as memory unless you built the memory. Grounding means the facts in an answer come from evidence you provided, and it changes the design by forcing you to keep source ids through the pipeline so they can be shown and checked.

Structured output validation

Whenever model output feeds code, define a schema and validate it, treating a parse failure as an expected event with a retry-or-route policy. This is what makes Pattern 1 possible and what keeps tool arguments in Pattern 5 within bounds. The schema checks shape; it does not check truth, which is why arithmetic checks, enum membership and authorisation are separate steps.

Confidence, abstention and fallbacks

A system that always answers is a system that sometimes answers confidently and wrongly. Give every pattern a defined way to decline: retrieval score below threshold, schema validation failure, a failed rule, or a classifier uncertain between labels. Declining must lead somewhere useful: a human queue, a search page, or a form. Fallbacks follow the same logic for infrastructure: when the primary model times out, degrade to a smaller model, a cached answer or plain keyword search, and record which path served the request. Retry and backoff apply to model calls as they do to any dependency, with the same risk of retry storms (see Retry & Backoff).

The flow below traces every exit of the call sketched in the pseudocode that follows it.

async function guardedCall<T>(req: Request, spec: Spec<T>): Promise<Result<T>> {
  const span = trace.start(spec.name, { feature_flag: flags.variant(spec.name, req.user) });
  const key = cacheKey(spec.name, spec.version, normalize(req.input));   // include prompt+model version
  const hit = await cache.get(key);
  if (hit && spec.cacheable) return span.end(hit, { path: 'cache' });

  const ctx = buildContext(req, spec.tokenBudget);                       // ordered, budgeted, with provenance
  for (const model of [spec.primary, spec.fallback]) {
    try {
      const raw = await call(model, ctx, { timeoutMs: spec.timeoutMs });
      const parsed = spec.schema.safeParse(raw);
      if (!parsed.success) continue;                                     // try the next option
      if (!spec.verify(parsed.data, ctx)) return span.end(abstain('failed_verification'));
      if (spec.cacheable) await cache.set(key, parsed.data, spec.ttl);
      return span.end(parsed.data, { path: model.name, tokens: raw.usage });
    } catch (e) { span.event('model_error', { model: model.name }); }
  }
  return span.end(abstain('exhausted'));                                  // always a defined outcome
}

Observability, evaluation and flags

You cannot debug what you did not record. Trace every request with the assembled context version, retrieved ids, model and prompt version, tool calls, validation outcomes, latency per stage and token counts, with redaction for personal data. Traces answer “why did it say that?” and are the raw material for evaluation: a labelled set of real cases, run on every change, tracking task quality rather than only whether the JSON parsed. Feature flags tie these together by letting you ship a new prompt, model or retrieval setting to a small share of traffic, compare it with the baseline, and roll back without a deploy. See AI Observability for the platform side.

Caching and cost controls

Caching changes the design because an exact-match cache is cheap and safe for deterministic steps (a classification of an identical email), while a semantic cache can return a similar but wrong answer, and a cache shared across users can leak private data unless the key includes tenant and permissions. Provider-side prompt caching rewards a stable prefix, so put the static instructions first. Cost controls are design inputs: a token budget per context, a maximum number of calls per request, a spend limit per agent run, routing easy cases to cheaper models, and a dashboard of cost per successful task, not cost per call. Details are in Caching in AI Systems and Cost Optimization.

The tree and axes only help when applied. Below are five requests as a product manager might phrase them, each reduced to the properties that matter, with a recommended minimal architecture. The numbers are illustrative.

1. “Tag and route incoming support emails”

Input is free text, the output is one label from a closed set (billing, shipping, technical, account), freshness is irrelevant, and an occasional wrong label is tolerable if it can be corrected cheaply. There are no side effects beyond routing, and no human control is needed for confident cases. Scale is 5,000 emails per day, about 0.06 per second on average, so even a peak of ten times that is a handful of requests per second. The minimal design is Pattern 1: a classifier (a small model, or an LLM with a constrained label enum), a threshold below which the email goes to a triage queue, and code that maps labels to queues. No retrieval, no agent, no chat.

2. “Let employees ask about the travel policy”

Input is natural language; the source is a document the company owns; freshness means “updated when the policy changes,” perhaps a few times a year; correctness tolerance is moderate, but users act on the answer. No side effects. The tempting answer is a full RAG stack. But if the policy is 30 pages at roughly 500 tokens per page, that is 15,000 tokens, which fits many context windows, so the minimal first version puts the document in the prompt with section ids and demands cited sections, with an abstention instruction and a verifier. Retrieval is added when the corpus outgrows that, when ACLs differ by audience, or when cost per question demands it. Starting simple gives you the evaluation set and the citation UX you will need anyway when you do add Pattern 2.

3. “Draft replies for our sales reps”

Input is an email thread plus CRM fields; the output is free text; correctness tolerance is low because statements go to customers, but a rep reads every draft. The only side effect is sending, and it is performed by the person. That is a Pattern 3 copilot: assemble context from the thread and the CRM through authorised reads, generate a draft, show it with the facts it used, and let the rep edit and send. The design keeps the human as the sender; there is no scenario where “auto-send” becomes a flag for the model to flip.

4. “Personalise the homepage shelf for 2 million daily users”

Input is behavioural events about items; the output is an ordering; freshness matters on the order of minutes to hours; correctness is statistical, with no single right answer; scale is high and latency is tens of milliseconds. This is Pattern 4, with candidate sources, a ranker and diversity rules. A generative model has no role in the scoring path. At most, it could write a human-readable shelf title offline, in batch.

5. “Automatically rebook passengers after a cancellation”

Input is a structured event; the side effects are high (money, seats, notifications), so correctness tolerance is very low and users cannot easily undo a wrong rebooking at an airport. Scale is bursty: a storm can cancel thousands of flights in an hour. Most cases follow a known path: find the next compatible flight, check entitlement, rebook. So the minimal design is a deterministic workflow with idempotent steps and a queue, plus a model only for the exceptions, such as a passenger with a special assistance request that needs free-text interpretation, which goes to a bounded agent or a human agent proposing options for approval. Everything that books a seat is executed by code under policy, not chosen by a model.

Notice what the five answers share: each states the smallest component set that meets the properties, and each names the thing deliberately left out. The table that follows only restates the outcome of that reasoning.

Real products rarely use one pattern. A customer support application that answers policy questions, looks up account data and issues refunds uses four of them, and the architectural risk moves to the seams. The case-level behaviour is the subject of AI Customer Support Systems, and the organisational view in Enterprise AI Applications; here the focus is on interfaces and ownership.

Follow the interfaces. The orchestrator hands the policy retriever a query and the user’s identity, and gets back passages with ids, never a decision. It hands the account service a customer-scoped request, and the service enforces that scope itself, so a confused or manipulated model cannot ask for someone else’s order. It hands the approval queue a proposal, which includes the policy clause id and the order facts the approver needs. Payments are executed only after approval, with an idempotency key. At no point does model-generated text become an authorisation: text that arrives from a customer or a retrieved document is data, and an instruction hidden inside it, a prompt injection, has no power because the tools do not trust the model’s say-so (see AI Security).

{
  "proposal_id": "pr_3310",
  "case_id": "CASE-8120",
  "action": "refund",
  "charge_id": "ch_5521",
  "amount_cents": 12900,
  "currency": "USD",
  "basis": { "policy_clause": "RET-DAMAGE@v2#cl_7", "order_day": 9, "evidence": ["photo_up_881"] },
  "idempotency_key": "CASE-8120:ch_5521",
  "requires": "human_approval",
  "trace_id": "tr_a91f"
}

Ownership boundaries decide who is paged when something breaks. A wrong policy answer is a knowledge-base and retrieval problem, owned by the content and search team; a wrong order total is an account-service defect; a wrongly approved refund is a process and review-quality problem; and a confused plan is the orchestrator team’s prompt and flow. The trace ties them together, because each component records the ids it received and returned. Without that, an incident review becomes a debate about whose part failed.

Most AI product failures that teams regret are failures of excess, not of missing capability. Each one below has a recognisable symptom and a simpler design that avoids it.

Unnecessary agents. The symptom is a trace that visits the same tools in the same order for 95 percent of cases (illustratively); the cure is the fixed workflow of Pattern 5, with an agent reserved for the residual cases. Excessive model calls. A planner, a retriever agent, a writer and a critic for a question whose answer is a row in a table. Unverified citations. Showing source links that the model chose, with nobody checking that the passage supports the claim, creates a false sense of authority worse than no citation; the simpler design verifies that ids exist in the context and samples support checks. Stale memory. A stored “user prefers metric units” that has been wrong for a year; keep durable facts in an explicit store with source, timestamp and user-visible edit, or do not store them. Unbounded context. Appending every turn and every tool result until cost and errors rise; budget the context. Poor evaluation. Judging by a handful of demos; keep a labelled set from real cases and run it on every change. Opaque UX. A user cannot tell what the system used, what it did, or how to correct it; show sources, show proposed actions before they happen, and offer an undo or a handoff.

The anti-overengineering case: a company builds a “multi-agent” FAQ bot for the question “What are your opening hours?” A router agent, a planner, a retriever, a writer, a critic and a formatter run as nine sequential model calls (illustratively 1.5 s each, so 13.5 s). If each call has an independent 98 percent chance of behaving as intended (an assumption, not a measurement), the chance that all nine do is 0.98⁹ ≈ 0.834: about one request in six goes wrong somewhere in the chain. The answer exists in a structured table; a lookup returns it in milliseconds and cannot be wrong in this way. A model enters only for the remainder: questions that do not match any table row, answered by Pattern 2 with a single call. The compounding-error arithmetic applies to any serial chain, which is why every extra call has to buy a measured improvement.

You rarely know the right pattern up front, so build the answer incrementally and let evidence add components. Start with a baseline that is as simple as you can defend: for the invoice case that is Pattern 1 with a basic prompt and the validators. Build a labelled evaluation set from real cases before tuning anything. Then run the baseline, and read the errors.

baseline: 261 / 300 fully correct = 87.0 %
errors (39):
  17  date formats (e.g. 03/04/26 read as the wrong day-month order)
  12  multi-line descriptions merged into one line item
   6  currency symbol ambiguity
   4  other / unreadable scans

candidate fixes, cheapest first:
  dates     -> normalise in code with the vendor's locale          (no model change)
  lines     -> layout-aware table extraction                       (new component)
  currency  -> enum in the schema + currency validator             (schema change)
  other     -> leave to clerk review

if the first three fixes work: 261 + 17 + 12 + 6 = 296 / 300 = 98.7 %

Every change then passes through the same controlled evaluation, run against the labelled set so that you can tell whether a fix that improved dates broke something else. Beyond offline numbers, stage the deployment: run the new variant in shadow mode, where it processes real traffic but its output is only logged; then route a small share, say 5 percent, behind a feature flag; compare against the baseline; and widen to 25 and 100 percent only if the metrics hold. Keep the old path available so that rollback is a flag flip.

Track three families of metrics together, since optimising one in isolation causes damage. Task quality: for the invoice case, the share auto-posted correctly, the share routed to review that did not need it, and the error rate that escaped to the ledger. Latency: end-to-end per stage, at p50 and p95, since users feel the tail. Cost: per successfully completed task, including retries and human review time, because a cheap model that triggers more review can raise the real cost. Add components when error analysis demonstrates a need and remove them when they stop earning their keep. See Production AI Architecture for how these controls sit in a larger platform.

Try each brief yourself before reading the reference reasoning. All three are intentionally under-specified, as real requests are, so the first job is to ask the questions that the axes suggest.

Brief A: “Use AI to flag duplicate expense claims”

The questions: what makes two claims duplicates, how are they entered and what does a false flag cost? The input is structured (employee, merchant, amount, date, receipt number). A duplicate is the same employee, a near-identical merchant, the same amount and a date within a few days, or the same receipt number. All of this is rules. The reference solution needs no generative AI: normalise merchant names, key on the tuple, use a fuzzy string distance for merchant variants and an exact match on receipt number, and send matches to a reviewer with the matching fields highlighted. If measured misses show that merchant names are too varied for fuzzy matching, only then consider a model, and only to normalise names, with the matching still done by code.

Brief B: “Turn meeting recordings into Jira tickets”

The input is audio, the work is minutes long, and the side effect is ticket creation in a shared system. The reference solution is an asynchronous pipeline (Pattern 6 mechanics): transcribe the audio, then extract action items with a strict schema containing owner, due date and the quoted transcript span as evidence (Pattern 1), then show them to the meeting organiser, who edits and confirms (Pattern 3). Only confirmed items are created, with an idempotency key per item so re-running the job cannot create duplicates. This is not an agent: the path is known and the creation step needs a human, because owners and due dates are exactly the details a model may infer wrongly.

Brief C: “Help on-call engineers find the cause of incidents”

Here the path really is uncertain: the next query depends on what the previous one revealed. Freshness is critical (current metrics, recent deploys), so a static index is insufficient. Side effects must be zero in the first version. The reference solution is a bounded agent with read-only tools (metrics query, deploy history, log search with scoped access) plus retrieval over runbooks (Pattern 2), a step and time budget, and output that is a ranked list of hypotheses each linked to the evidence it was based on. Remediation (a restart, a rollback) stays a human action. Only after months of traces show that a specific remediation is safe and repetitive would you promote that one step into a fixed, approved, automated workflow.

1. 1

   Choose the pattern from the job, not the model: mark each step as known, fuzzy input or fuzzy path, and ask the cheapest question first. A fixed invoice workflow does not need an agent choosing its next step.

2. 2

   Deterministic code with the model at the fuzzy boundary: ask the model for what is printed, enforce the schema at the interface, and let validators catch arithmetic mismatches.

3. 3

   Retrieval-grounded answers need identity-scoped retrieval, stable chunk IDs, citations, a score threshold below which the system declines, and an ingestion process that retires superseded versions.

4. 4

   Copilots keep the decision with a human, with evidence attached to each suggestion and accept, edit and reject rates tracked per rule. Rankers use candidate retrieval then ranking, and agents get tools with permissions, idempotency and read-back verification.

5. 5

   Slow generation belongs in an asynchronous pipeline with an idempotency key per job and moderation as a stage with authority to block.

6. 6

   Most regretted failures are excess, such as unnecessary agents. Start from a simple baseline, add components only on error-analysis evidence, and roll out in stages.

Look back at the model boundaries in the six patterns. The extractor of Pattern 1 needs instructions that define the schema and say what to do with a blurry field. The answer writer of Pattern 2 needs instructions to use only the supplied passages and to cite their ids. The suggestion generator of Pattern 3 needs a playbook in its context, the tool-using model of Pattern 5 needs descriptions that help it pick the right tool, and the classifier in the routing example needs a label list. Every one of these is written in natural language, and the quality of that language measurably changes how often each boundary behaves as designed.

There is a limit that this lesson has kept returning to. Instructions shape what a model is likely to do; they do not enforce authorisation, tenant isolation, an approval gate or a rounding rule. Those live in code, which is why the patterns put validators, ACL filters, permission checks and idempotency keys around the prompts rather than inside them. The next lesson, prompt engineering, studies the instruction layer in its own right: how the same model, given a different prompt, can produce very different results, and how to design, test and version prompts so that each boundary in your architecture is as reliable as it can be.