Production Agent Architecture

Watch one request move through planning, retrieval, tool use, approvals, and observability so the final answer is grounded, governed, and operable.

What changes when an AI agent goes to production is everything around the model: durable state so runs survive crashes, tool authorization enforced outside the model, evaluation before release, observability of every step, and a rollback path for prompts, tools and policy. A production AI agent architecture is therefore not a better prompt around a model. It is a durable, multi-tenant workflow system in which model calls are one kind of step. The model proposes; trusted code validates, authorizes, executes and verifies. Every step is persisted before and after it happens, so a crash, a timeout or a duplicate delivery resumes from known state instead of starting over. Side effects carry idempotency keys and are reconciled when their outcome is unknown, humans approve exact actions without holding a worker hostage, and every decision leaves a trace an on-call engineer can read. Quality is gated by evaluation before release and measured by verified outcomes afterwards, and the whole thing is sized by token and latency arithmetic rather than hope.

This lesson builds that system around one running case and then makes it fail on purpose. The case is the refund agent from the earlier lessons: "Order 8472 was charged twice, and my refund has not arrived. Please investigate and resolve it." The internals of each part (the loop, tool calling, planning, guardrails) have their own lessons and are linked where they matter; here the subject is how the parts behave together when things go wrong. All identifiers, latencies, token counts, prices and thresholds below are invented for illustration; none are measurements.

The prototype is a single process: a loop that calls a model, executes the tool call the model emits, appends the result to a message list in memory, and repeats until the model writes an answer. On a developer's laptop, against a sandbox payments API, with one request at a time, it investigates order 8472 beautifully. It finds two captures of ₹2,000, reads the refund policy, submits a refund and writes a polite reply. Everything that makes the demo look finished is also what the first week of production dismantles.

Two requests arrive for order 8472. The customer pressed send twice, and the mobile client retried after a flaky connection. The prototype has no notion of a task identity, so two loops run in parallel, both read "no refund exists yet", and both submit one. The customer is refunded ₹4,000 for a ₹2,000 duplicate charge. The model did nothing wrong; two uncoordinated runtimes each did something reasonable.

The model provider returns errors for ninety seconds. The prototype raises an exception mid-task. The tool call that already ran is lost with the process's memory, and the customer who retries triggers the whole investigation, including its side effects, again.

The refund needs a human. Policy requires approval for refunds of ₹1,000 or more. The prototype blocks on a input()-style wait inside the loop. The approver is in a meeting; the worker is pinned for forty minutes, the deployment restarts for an unrelated reason, and the pending approval, which lived only in a stack frame, disappears. When the approver clicks "approve" a minute later there is nothing to approve.

A payments call times out after sending the request. The prototype treats a timeout as failure and retries. The first request had in fact succeeded. This is the failure the rest of the lesson keeps returning to, because it is the one where doing the obvious thing costs money.

Tenants share a process. One business customer's retrieval query is cached keyed by the question text alone, and a similar question from another tenant gets a passage it should never see. Costs climb because every new tool result is re-sent on every later model call, so tokens grow faster than the number of steps. Nobody can say why, because the only record is a log line per model call.

None of these is a model-quality problem. They are the problems every backend engineer has met in payment processing, job scheduling and workflow systems: identity, concurrency, durability, at-least-once delivery, long waits, isolation, observability and capacity. The agent's new element is that one of its steps is non-deterministic and persuasive. That raises the stakes for the old solutions rather than replacing them. The architecture that follows is what you get by applying them with the model treated as an untrusted proposer (see Agent Components for what that runtime contains and The Agentic Loop for the cycle it runs).

Architecture follows from requirements, and agent requirements are easy to under-specify because the demo has only one: "it works". Writing them as measurable properties is what lets later sections justify each component. The targets below are illustrative for a refund agent; yours will come from your business and regulators.

Two requirements deserve more explanation than a row in a table. First, cost per successful taskrather than cost per request: a failed or abandoned task still spent tokens, so the denominator is tasks that reached a verified correct outcome. Second, approval SLA is a property of the human process and must be measured separately from system latency. If approvals take nine minutes, the system's p95 is irrelevant to the customer, but the system must make that wait cheap: no held worker, no held connection, no lost state.

The architecture splits by a distinction familiar from networking and Kubernetes. The control plane decides and records: who may do what, which versions of prompts, tools and policy are live, which approvals exist, what the evaluation suite says about a candidate release. It changes slowly, is audited heavily and must be correct. The data plane does the work for each task: it dequeues, calls models, runs tools, retrieves evidence and writes checkpoints. It scales with traffic and must be fast and resilient. Keeping them separate means a spike in refund requests does not starve the approval service, and a bad prompt release can be reverted in the registry without touching running workers.

The figure below shows the production AI agent architecture as two planes with the model providers, document indexes and business systems outside the platform boundary. For the single-agent runtime that sits inside the data plane, see Single-Agent Architecture.

Walking the figure for order 8472. The API gateway authenticates the caller and attaches a tenant identity that every downstream component treats as authoritative; nothing the model or the user writes can change it. The task service turns the request into a durable task row and an enqueue event in one transaction (the arrow along the left edge of the data plane). Orchestrator workerslease tasks from the queue and execute them step by step against the checkpoint database. When a step needs inference they call the model gateway, which owns provider credentials, version pinning, routing and budgets. When a step needs the outside world they go through the tool executor: it validates arguments against the tool's schema from the registry, asks the policy engine whether this exact action is allowed, and then uses per-tool credentials to call the adapter, which may be an MCP client talking to an MCP server (Model Context Protocol). Evidence retrieval is just a read-only tool served by the retrieval service, with the same authorization checks, so the document index is not a back door.

Three properties of this layout matter more than the box count. First, the orchestrator holds no credentials; secrets live behind the tool executor and model gateway, so compromising the part that talks to the model does not hand over the payments API. Second, policy and approval are trusted code outside the model's reach; a model can ask, never grant (see HITL and Guardrailsfor the guardrail mechanics). Third, every component emits spans, metrics and audit recordsto the observability plane; it is drawn at the bottom because it is not on the request path but is the only reason the others can be operated.

The customer's message is the same request every time; the system's job is to turn it into a task with an identity and then process that task asynchronously. The contract at the edge is the first place durability shows up. A synchronous "send message, wait for the answer" endpoint would hold a connection open across model calls and a human approval. Instead the client submits the work and gets a handle.

POST /v1/tasks
Authorization: Bearer <jwt>                       # tenant and subject derived from the token
Idempotency-Key: req-8472                         # the client's request identity
Content-Type: application/json

{ "message": "Order 8472 was charged twice, and my refund has not arrived. Please investigate and resolve it.",
  "channel": "chat" }

202 Accepted
Location: /v1/tasks/tsk_8472_a
{ "task_id": "tsk_8472_a", "status": "RECEIVED", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736" }

The tenant is taken from the verified token, never from the request body. The Idempotency-Keymakes retries safe: the task service has a unique constraint on (tenant_id, idempotency_key), so the duplicate request from the double-tap resolves to the existing task. If the same key arrives with a different payload the service rejects it with an error rather than guessing. Following the request:

1. 1

   Authenticate, rate limit, create

   The gateway validates the token and applies the tenant's rate limit. The task service inserts the task row and an outbox event in one transaction, returns 202 and the task id. From here on the request is durable.
2. 2

   Lease and plan

   A worker leases tsk_8472_a (30 s TTL, renewed by heartbeat), loads state and asks the model gateway for the first decision. The model proposes two read-only calls: fetch the order and list its payments.
3. 3

   Evidence from authorized systems

   The tool executor validates both calls, authorizes them under the caller's tenant, and runs them in parallel. The observations, not the model's guesses, establish that payments pay_8472_a and pay_8472_b both captured ₹2,000 for the same order.
4. 4

   Policy grounding

   The retrieval service returns passages from POL-REF-004 v4 (illustrative) filtered to the tenant's authorized collections. Retrieved text is data for the model to read, not an instruction channel (see the Context Injection lesson).
5. 5

   Proposal, then gate

   The model proposes refunds.create for pay_8472_b, ₹2,000. The policy gate in trusted code evaluates the proposal: duplicate capture is confirmed by the ledger observation, amount is under the order total, and amounts of ₹1,000 or more need approval. Decision: needs_approval.
6. 6

   Park and checkpoint

   The exact action and its hash are written to an approval record and the task checkpoints in APPROVAL_WAIT. No worker, thread or connection waits. The status endpoint now reports awaiting_approval.
7. 7

   Resume and revalidate

   When the approver acts, an event wakes the task. The orchestrator re-checks the grant (same action hash, not expired, not used), the current policy version, and that the pending action still passes the gate before anything executes.
8. 8

   Execute with an idempotency key

   The intent and key are checkpointed first, then the executor calls the payments API with that key. The outcome is recorded either way.
9. 9

   Verify, then respond

   Deterministic code checks the refund against the ledger. Only verified facts may appear in the reply; the model drafts it, a claim check validates it, and the task reaches COMPLETED with the final message attached.

The client learns about progress by polling GET /v1/tasks/tsk_8472_a or by a webhook/event, and can cancel with POST /v1/tasks/tsk_8472_a/cancel. Cancellation is a flag the worker honors at the next step boundary; a cancel that arrives while a refund is in flight cannot recall it, which is why the status distinguishes "cancel requested" from "cancelled" and why the reconciliation path exists.

If a process can die between any two statements, the only way to be correct is to decide in advance what is written to durable storage and when. For an agent task the persisted state is: the task and its status; the ordered history of model outputs and tool observations (so a resumed task sees exactly what the first run saw, without re-asking the model); pending actions and their action hashes; approval records; retry counters and next-attempt times; the versions of prompt, policy and workflow in force; and the terminal outcome. These are stored as an append-only event log plus a current-state row updated by compare-and-swap on a version number.

The state machine is small on purpose. APPROVAL_WAIT is a state, not a thread, which is why a nine-minute approval costs nothing. EXECUTING means "intent has been recorded and the call is, or may be, in flight". EXEC_UNKNOWN exists because a timeout after sending a request is not a failure; it is an unknown, and the only exits from it are by looking at the external system. Collapsing it into FAILED would invite a blind retry, and collapsing it into EXECUTING would hide that the answer is uncertain.

{
  "task_id": "tsk_8472_a",
  "version": 12,
  "status": "EXECUTING",
  "versions": { "prompt": "refund-v7", "policy": "POL-REF-004@4", "workflow": "refund_resolution@3", "model_route": "support-tier2@2026-03" },
  "evidence": [
    { "id": "obs_3", "tool": "payments.list",  "result_ref": "blob://obs/tsk_8472_a/3", "trust": "system_of_record" },
    { "id": "obs_5", "tool": "retrieval.search", "doc": "POL-REF-004#c0003", "doc_version": 4, "trust": "untrusted_text" }
  ],
  "pending_action": {
    "tool": "refunds.create",
    "args": { "payment_id": "pay_8472_b", "amount_minor": 200000, "currency": "INR" },
    "action_hash": "sha256:a91c4e07…",
    "idempotency_key": "refund:req-8472:pay_8472_b",
    "approval_id": "apr_5521",
    "approval_consumed_at": "2026-03-14T10:11:17.249Z"
  },
  "lease": { "worker": "w-07", "expires_at": "2026-03-14T10:11:47.259Z" }
}

Note the idempotency key. It is derived from the identity of the action (the request and the payment being refunded), not from the attempt, so the first try and every later try present the same key to the payments API. If the payments API supports idempotent creation, a repeated call returns the original result instead of creating a second refund.

Exactly-once effects are built, not delivered

Queues, workflow engines and HTTP clients do not provide exactly-once delivery across a network that can lose a response; what you can have is at-least-once execution with an effect that is made idempotent, or an unknown outcome that is resolved by reconciliation. The workflow engine Temporal states this posture for its Activities: it recommends they be idempotent, because an Activity that fails to report its result is retried, while an Activity that has completed is not re-executed when the workflow replays (documented in Temporal's Activity definition guide). LangGraph, a graph-based agent framework, takes the complementary angle for state: its persistence layer saves a thread's graph state as checkpoints, keyed by a thread_id, which it describes as enabling human-in-the-loop workflows and fault tolerance, and its documentation advises a persistent checkpointer rather than the in-memory one for production. Neither product removes the need to make your own side effects safe to repeat; that part is yours, and these are version-specific features to recheck against current documentation before you depend on them.

The practical rules follow. Write intent before acting. Give each external effect a stable idempotency key. Classify every tool as having no side effect, an idempotent one, or a non-idempotent one, and give the third class a reconcile function that can answer "did this happen?" from the external system's own records. When the outcome is unknown, reconcile first and retry second. The incident walk-through later in the lesson runs this rule against a real timeout.

Task processing is asynchronous because its duration is unbounded (approval waits) and its dependencies are slow (model calls dominate, as the cost section will compute). A queue between the task service and the workers turns bursts into lag instead of failures, and lets workers scale by the one number that matters, queue age.

The topology has three jobs beyond "put tasks in a line". Fairness: one tenant importing a batch of 40 tasks must not push another tenant's single urgent task to the back, so queues are partitioned by tenant and a scheduler takes from them round-robin, with a per-tenant token bucket at admission that answers 429 with Retry-After when the bucket is empty. Backpressure: the real bottleneck is usually not worker CPU but the model provider's rate limit or your own concurrency cap, so the cap belongs in the model gateway and the scheduler should slow dispatch when it is saturated. Recovery:workers hold a lease on a task and renew it by heartbeat; if a worker dies the lease expires and another worker resumes the task from its last checkpoint. Duplicate delivery of the same queue message is harmless because the second worker either fails to acquire the lease or fails its compare-and-swap commit.

Dead-letter handling needs care because the payload is rarely the problem. A task that exhausts retries moves to a dead-letter queue with its checkpoint intact, raises an alert, and is replayable after the cause is fixed. For tasks that reached EXEC_UNKNOWN, the runbook is reconciliation, not replay. Resume events (approval granted, timer due, cancel requested) enter the same queue so there is one way to wake a task.

Sizing the cap: Little's law with the numbers from the cost section

Suppose a tenant mix produces 5 new tasks per second at peak. Later we will compute that each task spends 9.434 s inside model calls in the baseline design (illustrative). By Little's law, average concurrency equals arrival rate times time in system, so 5 × 9.434 ≈ 47 model calls in flight. If the gateway's concurrency cap is 40, the platform cannot sustain that rate: the sustainable throughput at the cap is 40 ÷ 9.434 ≈ 4.24 tasks per second, so lag grows without bound. Trimming one model call per task (7.874 s of model time) lifts it to 40 ÷ 7.874 ≈ 5.08 tasks per second. The queue and quotas do not make the work cheaper; they make overload visible, bounded and fair, and the call-count work in the latency section is what actually adds capacity.

Change the arrival rate, the cap and the design in the figure below to see when the queue stops keeping up.

When a synchronous process is still enough

Not every agent needs this machinery. A read-only assistant that answers from retrieval, finishes in a few seconds, has no side effects and loses nothing if a request is simply retried can run as a synchronous request handler, with a timeout, rate limiting and tracing. The threshold for introducing queues and durable state is crossed by any of: side effects that must not repeat, waits on humans or slow systems, tasks longer than a request timeout, or a need to resume after deployment. This is also the first rollout phase later in the lesson, and it is a feature, not a shortcut, to ship it that way.

Calling a model provider from every worker with its own SDK and keys scatters concerns that need one owner. The model gateway is a thin service that every inference call passes through, and it is where model non-determinism meets operational discipline.

route: support-tier2
  primary:   { provider: providerA, model: model-x-2026-03, pinned: true }    # exact snapshot, never "latest"
  fallback:  { provider: providerB, model: model-y-2026-02, allowed_for: [summarize, draft_reply] }
  timeouts:  { connect_ms: 1000, first_token_ms: 4000, total_ms: 20000 }
  breaker:   { open_after_failures: 5, window_s: 30, half_open_probe_s: 20 }
  budgets:   { max_input_tokens_per_call: 12000, max_calls_per_task: 8, max_input_tokens_per_task: 60000 }
  output:    { schema: proposal_v3, on_invalid: repair_once_then_fail }
  cache:     { scope: tenant, applies_to: [static_prefix], never: [personal_data_in_key] }

Version pinning means requests name an exact model snapshot, so a provider-side alias change cannot silently alter behavior; promotions go through the registry and the evaluation gate described later. Timeouts and circuit breakers keep a slow provider from consuming the whole worker pool: when five calls fail in thirty seconds the breaker opens, calls fail fast, and tasks move to RETRY_WAITwith backoff rather than piling up. Token budgets are enforced here because the gateway can count: a per-call cap protects the context window (see the Context Window lesson), and a per-task call and token cap is the circuit that stops a looping agent from spending without bound.

Fallback is not a free redundancy. A different model, or even a different snapshot of the same model, can follow the schema differently, choose different tools, and disagree about borderline cases. The config above therefore restricts the fallback to low-risk step types and runs the same output-schema validation on every response. Proposals from a fallback model still pass through the same policy gate and, for refunds, the same human approval; the trace records which model produced each proposal so a quality dip after an outage can be traced to it. Structured-output validation exists because model output is text until proven otherwise: the gateway validates against the schema, may attempt one automatic repair, and otherwise fails the step into the normal retry path instead of passing malformed JSON to a tool.

Caching has constraints. Exact-match caching of a deterministic, non-personal call is safe. Provider-side prompt caching rewards a stable prefix (system prompt, tool schemas) and is why the config marks only the static prefix; the specific discounts and eligibility rules are provider-specific and must be checked against current provider documentation. Caching a model answer keyed on a similar question across tenants, or any response containing personal data, is a data-isolation bug waiting to happen, so cache scope is tenant and the cache key never contains personal data.

Tools and Tool Calling explains how a model's tool-call proposal becomes an invocation. The production concern is the layer that makes every such invocation safe and auditable regardless of which tool it is. That layer is the tool executor, and it enforces, in trusted code and in this order: discover the tool from a registry (the model sees only tools its task is entitled to); validate arguments against the tool's JSON schema; authorize the specific call against policy using the caller's tenant and the task's scope; mint a short-lived credential scoped to this tool; execute with the tool's timeout and idempotency key; classify the outcome as ok, rejected or unknown; and record the call in the audit log.

Business authorization stays in trusted code because it depends on facts a model cannot be trusted with. "Refunds may not exceed the captured amount of the payment being refunded" is checked from the payments ledger, not from what the model says the ledger contains. The registry entry records the facts the executor needs:

{
  "name": "refunds.create",
  "version": "2",
  "side_effect": "non_idempotent_unless_keyed",
  "args_schema": "schemas/refunds.create.v2.json",
  "acl": { "roles": ["support-agent-runtime"], "max_amount_minor_without_approval": 100000 },
  "timeout_ms": 5000,
  "idempotency": { "header": "Idempotency-Key", "key_template": "refund:{request_id}:{payment_id}" },
  "reconcile": { "tool": "refunds.get_by_key", "timeout_ms": 3000 },
  "audit": { "log_args": "redacted", "retention_days": 540 },
  "server": { "type": "http", "trust": "first_party" }
}

When tools are exposed through MCP, the same executor sits in front of the MCP client. MCP standardizes how a client discovers and invokes a server's tools; it does not make a server trustworthy, does not authorize business actions, and does not by itself provide idempotency. The platform therefore treats each MCP server as a dependency with a trust level: first-party servers on the internal network are different from a third-party server whose tool descriptions and outputs are untrusted text. Descriptions from untrusted servers are reviewed or pinned rather than auto-trusted, credentials are scoped per server so one server never receives another's secrets, and tool outputs are tagged as untrusted observations, exactly as in the MCP lesson.

Two different needs hide under "memory". Grounding supplies the facts and rules the model needs for this task; memory carries information across tasks. The refund agent needs the first and almost none of the second.

Grounding splits by the kind of fact. Transactional facts (was the customer charged twice? what is the refund status?) come only from authorized tool lookups; no retrieved document can establish them, and a model that "remembers" a charge is hallucinating (see Hallucinations). Policy and procedure come from retrieval over authorized documents (the retrieval pipeline), with provenance on every passage: document id, version, retrieval time. Provenance is what lets the final answer cite POL-REF-004 v4 and what lets an engineer discover that a stale chunk of v3 was used. Freshness is handled in the index pipeline (re-index on policy publish, supersede old versions) and checked at use: the authorization decision never relies on retrieved text; it uses the policy engine's current version, and retrieval is for explaining the decision to the customer. Retrieved text enters the prompt as quoted evidence, as covered in Context Injection.

State for the conversation (what has been asked, what tools returned) lives in the task's event log and is re-assembled into the prompt each call; that is short-lived context, not long-term memory. When the context grows, compaction is a deliberate step with its own tests, not an accident of truncation. Structured memory, such as "this customer prefers email" or "case 7731 is open", belongs in a normal database with a schema, owner, retention policy and access controls, written by trusted code from verified facts.

Long-term memory is not automatically needed, and adding it has a real cost: every remembered item is personal data with a retention and deletion obligation, a possible source of stale or poisoned context, and a new way for one customer's information to leak into another's answer. Add it when an evaluated improvement in outcomes justifies those costs; until then, the refund agent can start every task from the authoritative systems and the current policy, which is also easier to audit.

A prompt cannot enforce a permission, a spending limit or a tenant boundary, because the prompt is input to the very component being constrained. Security in this architecture is structural: it is a property of which component holds which credential, which code path can reach which data, and which actions have no code path at all. The deployment-boundary figure in the blueprint section shows where those lines fall; this section explains what each one is for.

Tenant isolation starts with identity. The tenant id comes from the authenticated token and is stamped on every task, event, checkpoint, cache entry, vector query and tool credential request. Storage enforces it (tenant-scoped queries or row-level security, per-tenant encryption keys where the threat model calls for them) so an application bug cannot read across tenants unnoticed. Retrieval filters on the caller's authorizations before ranking, so unauthorized text is never in the candidate set, rather than being removed afterward.

Secrets and PII. Provider keys and tool credentials live in a vault and are accessed only by the model gateway and tool executor, which mint short-lived, per-tool tokens; the orchestrator, the prompt and the logs never contain them. A redactor sits in front of the model gateway so the model receives a customer reference rather than a name, email or card fragment; where the reply needs the real value it is re-inserted by a template after the model has finished, in trusted code. Third-party model providers therefore see the minimum needed, and traces record the redaction decision, as shown in the trace section.

Prompt injection and exfiltration. Anything the agent reads (an order note, a retrieved document, a tool result, another agent's message) is data that may contain instructions addressed to the model. Defenses that depend on the model resisting are mitigations, not controls. The controls are structural: the refund tool's schema has no destination-account field, so a model coerced into proposing "refund to account X" has no way to express it; the policy engine checks the amount and payment from the ledger; the agent has no general-purpose "fetch any URL" or "email anyone" tool through which data could leave; rendered output has links and images stripped or allow-listed; and an untrusted observation cannot raise the privilege of the task. The malicious-tool-output scenario below plays this out. HITL and Guardrailscovers the guardrail design; the point here is that these checks run on every deployment path, not just the one that was tested.

Sandboxing and least privilege. Anything that executes untrusted content (generated code, an unvetted MCP server) runs in a sandbox with no ambient credentials and an egress allow-list. Each task carries a scope, a list of tools and resources its workflow step may use, and the executor refuses anything outside it; the refund workflow's read steps cannot call refunds.create even if a model asks. Human approval is bound to one exact action: the grant stores the action hash, expires, is single-use, and the approver cannot be the requester, so it cannot be replayed against a modified proposal.

Audit and incident response. The append-only event log is the audit record: who or what proposed, which policy decided, who approved, what executed, what was verified. Retention length is set with compliance, and it must be reconciled with deletion obligations for personal data (store references and hashes where you can, and treat the details as a legal question rather than an engineering default). For incidents the platform needs levers that work without a deployment: a per-tool disable flag, a per-tenant downgrade to read-only, credential revocation, and the ability to list every task that used a given tool version or prompt in a time window. After containment, the offending case is added to the golden evaluation set so the same failure fails a build next time.

An architecture is credible only if it can say, for each realistic failure, how it is detected, what state survives, how it recovers and what the customer sees. The failures below are not hypothetical edge cases; each corresponds to a component in the reference architecture existing for exactly that reason. The side-effect timeout is summarised here and walked through step by step in the incident section.

When the customer writes back "my refund never arrived", the on-call engineer needs one thing: the history of task tsk_8472_a, in order, with durations and outcomes. A distributed trace gives that if every step is a span under one root, and the trace id is returned to the client at task creation and stamped on every log line, checkpoint and audit record. Model calls, tool calls, retrieval, policy checks, state transitions and approvals all become spans.

OpenTelemetry has a set of GenAI semantic conventions for exactly this. They define span kinds and attributes for model calls (operation chat), tool execution (execute_tool, with span name execute_tool followed by the tool name), retrieval, and agent or workflow invocation (invoke_agent,invoke_workflow), and attributes for requested model, token usage and tool call id. Two cautions from the current specification: the GenAI conventions are marked in development status, so attribute names can change and you should pin the version you instrument against; and the attributes that carry prompt, completion, retrieved-document and query content are opt-in and documented as potentially sensitive, so capturing them is a deliberate privacy decision. The excerpt below uses those names where they exist and ordinary custom attributes elsewhere; timings are the incident run narrated later.

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "root_start": "2026-03-14T10:02:11.000Z",
  "redaction": {
    "policy": "trace-redaction@5",
    "prompt_and_completion_content": "NOT_RECORDED",
    "tool_args": "sha256 only",
    "customer_fields": "[REDACTED:pii]"
  },
  "spans": [
    { "span_id": "00f067aa0ba902b7", "parent": null, "name": "invoke_agent refund-resolver",
      "start_ms": 0, "duration_ms": 579750, "status": "OK",
      "attrs": { "gen_ai.operation.name": "invoke_agent", "task.id": "tsk_8472_a", "tenant.id": "ten_41",
                 "workflow.version": "refund_resolution@3", "task.outcome": "COMPLETED" } },
    { "span_id": "a2fb4a1d1a96d312", "parent": "00f067aa0ba902b7", "name": "chat",
      "start_ms": 60, "duration_ms": 1430, "status": "OK",
      "attrs": { "gen_ai.operation.name": "chat", "gen_ai.request.model": "model-x-2026-03",
                 "gen_ai.usage.input_tokens": 3200, "gen_ai.usage.output_tokens": 120, "route": "support-tier2" } },
    { "span_id": "e457b5a2e4d86bd1", "parent": "00f067aa0ba902b7", "name": "execute_tool payments.list",
      "start_ms": 1490, "duration_ms": 300, "status": "OK",
      "attrs": { "gen_ai.tool.name": "payments.list", "gen_ai.tool.call.id": "call_02", "args_hash": "sha256:5e1c…",
                 "result.captured_count": 2, "customer.email": "[REDACTED:pii]" } },
    { "span_id": "9a2f6d0c71b3e450", "parent": "00f067aa0ba902b7", "name": "retrieval policy-kb",
      "start_ms": 3562, "duration_ms": 450, "status": "OK",
      "attrs": { "gen_ai.operation.name": "retrieval", "gen_ai.data_source.id": "policy-kb",
                 "doc": "POL-REF-004#c0003", "doc_version": 4, "authz_filter": "ten_41/policies" } },
    { "span_id": "3c19d85f20aa7b66", "parent": "00f067aa0ba902b7", "name": "policy.evaluate",
      "start_ms": 6159, "duration_ms": 40, "status": "OK",
      "attrs": { "policy.version": "POL-REF-004@4", "decision": "needs_approval", "action_hash": "sha256:a91c4e07…" } },
    { "span_id": "71d0be3a96c4f2a8", "parent": "00f067aa0ba902b7", "name": "approval.wait",
      "start_ms": 6199, "duration_ms": 540000, "status": "OK",
      "attrs": { "approval.id": "apr_5521", "worker_held": false, "approver": "usr_[HASHED]", "decision": "granted" } },
    { "span_id": "c8e2a07d5b9f1143", "parent": "00f067aa0ba902b7", "name": "execute_tool refunds.create",
      "start_ms": 546249, "duration_ms": 5000, "status": "ERROR",
      "attrs": { "gen_ai.tool.name": "refunds.create", "error.type": "timeout", "outcome": "unknown",
                 "idempotency_key": "refund:req-8472:pay_8472_b", "checkpoint": "ckpt_12" } },
    { "span_id": "5d4a1e9c0f72b8e3", "parent": "00f067aa0ba902b7", "name": "execute_tool refunds.get_by_key",
      "start_ms": 577000, "duration_ms": 180, "status": "OK",
      "attrs": { "gen_ai.tool.name": "refunds.get_by_key", "reconcile_for": "c8e2a07d5b9f1143",
                 "found": true, "refund_id": "rf_3301", "refund_status": "processing", "worker": "w-12" } },
    { "span_id": "0be7f93a4c6d2815", "parent": "00f067aa0ba902b7", "name": "chat",
      "start_ms": 577185, "duration_ms": 2525, "status": "OK",
      "attrs": { "gen_ai.request.model": "model-x-2026-03", "gen_ai.usage.input_tokens": 6150, "gen_ai.usage.output_tokens": 220 } }
  ]
}

Read it the way an on-call engineer would. The long bar is the approval wait, and worker_held: falsesays it cost no capacity. The one red span is the refund call, and its attributes say the outcome is unknown, point to the checkpoint that holds the intent, and carry the idempotency key. The gap between that span ending (546,249 + 5,000 = 551,249 ms) and the reconcile span starting (577,000 ms) is 25,751 ms, which is a crashed worker's lease running out and a new worker picking up the task, and it appears in the trace as a change of the worker attribute rather than as an error. The redaction boundary is explicit: content attributes are off, tool arguments are hashed so they can be correlated but not read, and customer fields are masked at the point the span is built, before it leaves the platform.

Spans answer "what happened on this task". Metrics answer "how is the fleet doing", and the same attributes become metric dimensions: tokens, calls and latency per stage, by route and version. Cardinality matters here: task ids and tool-call ids belong in traces, not as metric labels.

An agent change cannot be reviewed like a pure function, because a prompt edit or a model upgrade alters behavior across thousands of paths no diff shows. The release gate is therefore an evaluation pipeline that runs against a candidate release (a bundle of versions, covered in the change-management section) and returns pass or fail with evidence.

The foundation is a golden task set: curated customer situations with expected outcomes in the system's terms, not in prose. For the refund agent that means cases like: both payments captured (refund one); one capture plus a pending authorization hold that merely looks like a second charge (do not refund); already refunded (explain, do not refund again); refund of ₹2,000 needing approval; an amount above the captured value; a malicious note in the order; and a tool timeout after the side effect. Each runs against simulated tool environments that return scripted, deterministic responses and can inject faults (timeouts after success, duplicate deliveries, stale policy), so a failure is reproducible and the final state can be asserted: exactly one refund exists, for the right payment, with the right amount.

Scoring happens at several levels, each with a different kind of check. Deterministic contract testsneed no model at all: they verify that schemas reject bad arguments, the policy gate denies what it should, the approval grant binds to the action hash, the state machine rejects illegal transitions and the executor reconciles unknown outcomes. They are the fastest and most valuable tests in the suite because the safety-critical logic is ordinary code. Task-success scoring compares the final state of the simulated world to the expected one. Tool-call precision and recall asks whether the model proposed the needed reads and avoided unneeded or forbidden calls, which catches regressions that still end in a correct answer but waste cost. Safety and adversarial suites include prompt-injection attempts embedded in order notes and retrieved passages and assert that no unauthorized action executes; a single failure here blocks release. LLM-based or human grading is reserved for what cannot be asserted mechanically, such as the tone and completeness of the customer reply, and human review samples calibrate any automated grader rather than being replaced by it.

Because model output varies, run each case more than once for the cases that matter and report the variation; a model that gets a safety case right in four of five attempts has not passed it. Every model snapshot, prompt, tool-schema or retrieval-index change reruns the suite, and results are compared with the previous release, so a regression is a diff in a table, not a surprise in production. Evaluation does not prove correctness on unseen traffic; it narrows the risk to what the rollout phases below are designed to measure.

A running agent platform can be fully available and still be failing customers, so its service-level indicators are about outcomes as well as infrastructure. The ones that earn a dashboard are: the completion rate and, separately, the verified success rate (the outcome checked against the system of record); escalation rate; unauthorized-action attempts (proposals the gate denied) and unauthorized actions executed (which must be zero); p50, p95 and p99 active latency by stage; token cost per verified-resolved task; tool failure rate by tool; loop rate (tasks that hit the call cap); and drift in input mix or output distributions.

Each metric implies a remediation, which is the reason to track it. A rising loop rate with flat traffic usually points to a prompt or tool-schema change and is fixed by rollback and a new golden case. A rise in denied proposals means either a new injection campaign or a prompt regression that proposes forbidden actions; the denial reasons separate the two. A creeping cost per resolved task with unchanged call counts means larger observations or prompts, and the fix is compaction or truncation of tool results. Latency rising in one stage isolates the culprit: model p95 moving points to the provider or to longer prompts, retrieval p95 to the index, and approval age to the human process, which no engineering change will fix. Escalation up and verified success down after a model update is the signal that the fallback or upgrade changed behavior, and the trace's model attribute confirms it. Alerts follow SLO burn rates rather than single-sample thresholds, so a short provider blip does not page anyone while a slow bleed does.

Online evaluation closes the loop with the offline suite: sampled production tasks go to human review, disagreements become golden cases, and drift detectors compare today's input distribution with what the suite covers. That is how the set stays representative of the traffic you actually receive.

The performance of an agent is mostly the performance of repeated model calls, and the arithmetic is worth doing explicitly because the intuitive estimate (one call's cost times a few) is wrong in a specific way. Every call re-sends the entire accumulated context: system prompt, tool schemas, and every prior observation. So the input tokens billed grow with each step, and total input grows roughly with the square of the number of steps when each step adds context.

Assumptions
  latency per model call = 150 ms fixed + 0.1 ms per input token + 8 ms per output token
  invented unit prices (not any provider's): INR 0.20 per 1K input tokens, INR 0.80 per 1K output tokens
  static prefix (system prompt + tool schemas + output schema) = 3,000 tokens; task text = 200 tokens
  observations: orders 400 + payments 500, policy passages 1,200, refund result 250, verification 150

call  input tokens                           output  latency (ms)
 1    3,000 + 200                  = 3,200     120   150 + 320 + 960   = 1,430
 2    3,200 + 120 + 900            = 4,220     150   150 + 422 + 1,200 = 1,772
 3    4,220 + 150 + 1,200          = 5,570     180   150 + 557 + 1,440 = 2,147
 4    5,570 + 180 + 250            = 6,000     100   150 + 600 + 800   = 1,550
 5    6,000 + 100 + 150            = 6,250     220   150 + 625 + 1,760 = 2,535
total                                25,240     770                     9,434

Cost (baseline):  25.24 x 0.20 = 5.048   +   0.770 x 0.80 = 0.616   =  INR 5.664 per task
Amplification:    25,240 billed input tokens / 6,250 tokens in the final prompt = 4.04x

Add the tools and the plumbing to get the end-to-end budget. Ingress and task creation take 60 ms; the order and payments reads run in parallel, taking the slower of 120 ms and 300 ms, i.e. 300 ms; retrieval with reranking 450 ms; the policy gate 40 ms; the refund call 600 ms; the verification lookup 200 ms. Tool time is 300 + 450 + 40 + 600 + 200 = 1,590 ms, and the total active time is 60 + 9,434 + 1,590 = 11,084 ms, about 11.1 s, excluding the approval wait and about 80 ms of checkpoint commits (eight at 10 ms each). Model calls are 9,434 ÷ 11,084 ≈ 85% of the active time, so tuning anything else is second-order.

Reducing it, in the order that pays

Cut model calls that do not make decisions. Call 4 exists only to ask the model to request a verification lookup that the workflow always performs. Making verification a deterministic step removes the call. The final call now sees 5,570 + 180 + 250 + 150 = 6,150 input tokens, so total input is 3,200 + 4,220 + 5,570 + 6,150 = 19,140 and output 120 + 150 + 180 + 220 = 670. Cost: 19.14 × 0.20 = 3.828 for input plus 0.670 × 0.80 = 0.536 for output, INR 4.364. Model time becomes 1,430 + 1,772 + 2,147 + 2,525 = 7,874 ms (the last call is 150 + 615 + 1,760), and the total is 60 + 7,874 + 1,590 = 9,524 ms, 1,560 ms faster. The lesson is architectural: steps whose outcome is mandated by policy belong in workflow code, not in the model's loop.

Parallelize independent reads. Running orders and payments serially would cost 120 + 300 = 420 ms instead of 300 ms, 120 ms more. The saving is small here but larger when reads are slower, and the planner can emit independent calls together (see Planning and Task Decompositionfor which steps are independent). Cache the static prefix. If the provider supports prefix caching and the 3,000-token prefix is billed at an assumed 25% rate (INR 0.05 per 1K) on the four calls of the reduced flow, 12,000 tokens × (0.20 − 0.05) ÷ 1,000 = INR 1.80 is saved, and the cost falls to 4.364 − 1.80 = INR 2.564. The 25% figure is an assumption; check your provider's actual terms. Compress observations: a 400-token order record and a 500-token payment list contain fields the decision does not need, and every token removed from an early observation is saved on each later call.

Tier the models. Route steps by difficulty: a small model for classification and reply drafting from verified facts, a stronger one for the ambiguous-evidence decision. Tiering changes behavior, so it passes through the evaluation gate like any other change. Cap concurrency and calls. The cap on model calls per task (8 in the route above) bounds the worst case: with a base prompt of 3,200 tokens and about 800 tokens of new context per step, input tokens are N × 3,200 + 800 × N(N−1)/2, which is 24,000 at N = 5 and 68,000 at N = 10, so doubling the steps costs 2.83 times as much, and the number of calls matters more than any per-call saving. Batching helps non-interactive work (nightly re-evaluation of the golden set) but not the interactive refund path.

Drag the call count in the figure below to see the quadratic growth behind those numbers.

Finally, cost per successful task. If 90% of tasks reach a verified outcome (illustrative) and failed tasks spend as much as successful ones, the baseline costs 5.664 ÷ 0.90 = INR 6.29 per verified task, above the INR 4.00 target, and the reduced, cached design costs 2.564 ÷ 0.90 = INR 2.85, inside it. The difference came from three architectural decisions, not from a cheaper model.

A traditional service has one artifact version. An agent has at least seven things that change its behavior independently: the model snapshot and route, the prompts, the tool schemas, the workflow graph, the policy bundle, the retrieval index, and the evaluation suite that gates all of them. Changing any one can alter outcomes, so a release is a manifest that names them together, and the manifest is what gets evaluated, rolled out and rolled back.

{
  "release_id": "rel-2026-03-14-a",
  "model_route": "support-tier2@2026-03",
  "prompt": "refund-v7",
  "tool_schemas": { "refunds.create": "2", "payments.list": "5", "refunds.get_by_key": "1" },
  "workflow": "refund_resolution@3",
  "policy_bundle": "POL-REF-004@4",
  "retrieval_index": "policy-kb@2026-03-10",
  "eval_suite": "golden-refund@18",
  "rollout": { "mode": "canary", "traffic_percent": 5, "flags": { "auto_execute_under_minor": 0 } }
}

Shadow and canary. A candidate can first run in shadow: it processes copies of live tasks against read-only or mocked tools, its proposals are compared with the incumbent's and with human outcomes, and nothing it does reaches a customer. A canary then routes a small share of real traffic (say 5%, chosen by tenant so one customer's experience is consistent) with SLO guardrails that automatically revert on a breach of escalation rate, denied-proposal rate, verified-success rate or cost per task. Feature flags separate deploying code from enabling behavior, so the autonomous-execution threshold in the manifest can be set to zero in an incident without a deployment.

Rollback is only real if everything it needs is still there: the previous manifest, prompt and schemas must remain deployable, tool-schema changes must be additive within a major version, and persisted state must be readable by both versions. In-flight tasks are the hard part. A task started under workflow version 3 that is waiting nine minutes for approval must resume under something coherent. The straightforward rule is to pin each task to the workflow, prompt and route it began under and let new tasks use the new release, draining the old version when its in-flight tasks finish; the exception is policy, which is always evaluated at decision time and again on resume, because acting under a rescinded rule is worse than being inconsistent. Engines that replay recorded history to resume a workflow add their own constraints on changing workflow code while executions are in flight; if you use one, read its versioning guidance before planning migrations.

The decision is not "which is best" but which cost you would rather own, judged against the failures above. The dimensions that matter are durability (does a crash lose state, and who makes side effects safe), debuggability (can you reconstruct what happened from recorded history), security (who holds credentials, how are tenants isolated, can you audit), portability (what you must rewrite to leave), latency overhead, operational burden, and what your team already knows.

A custom loop with your own state tables is the right start for the read-only assistant: it is small, you understand every line, and its failure modes are your own. It becomes expensive when you reimplement leases, timers, approval signals, retries with backoff and versioning, because those are generic workflow problems with subtle edge cases. A general workflow engine (Temporal is a well-known example) supplies durable timers, retries, long waits and a recorded history, and treats a model call or tool call as an ordinary retried step; what it cannot supply is idempotency in your payments API, and you accept a service to operate and its constraints on workflow code. An agent framework such as LangGraph offers graph-shaped control flow, checkpointing and interrupts for human review, which speeds up building the agent logic; verify with current documentation which persistence and durability guarantees apply to your deployment mode, and note that framework abstractions and APIs change quickly, so isolating them behind your own ports (as the blueprint does) keeps a migration possible. A managed agent platformminimizes operations and time to first demo, and the questions become control: where data resides, whether tenant isolation and audit meet your obligations, whether you can export traces and state, how pricing scales with your cost-per-task model, and what happens when the product changes direction.

A defensible heuristic: if tasks are short, read-only and retryable, use the simplest thing. If they have side effects, long human waits or deadlines, use durable execution (an engine or equivalent) and keep your own authorization, policy and verification code regardless of what you adopt, since those are your business rules. Whatever you choose, demand the same properties from it: you can see every step, you can kill a task, you can replay a failure in a test, and the policy gate lives in code you control. Without those, no amount of convenience is worth the loss of debuggability.

The blueprint separates modules along the trust lines of the architecture and allows dependencies in one direction only: the orchestrator depends on small interfaces (ports), never on a database driver, a provider SDK or an MCP client, and the policy module imports nothing that can call a model.

agent-platform/
  api/           task API: create, status, cancel; authn; idempotency
  orchestrator/  lease, step loop, state transitions (talks only to ports)
  state/         CheckpointStore (SQL), append-only event log, outbox
  gateway/       ModelGateway: routes, pinning, budgets, schema validation
  tools/         ToolRegistry, ToolExecutor, adapters (http, mcp)
  policy/        PolicyEngine, ApprovalService   <- trusted code, no model imports
  verify/        claim verifier, reply renderer
  obs/           tracing, redaction, metrics
  evals/         golden tasks, simulated tools, scorers

The figure fixes what each boundary may carry. The orchestrator can reach the model gateway only through the redactor, can reach business systems only through the tool executor, and holds no credentials. The tool zone is the only place secrets are minted and the only network location with egress to business systems. The code below is illustrative TypeScript for the contracts that implement those lines; it is deliberately small and omits error handling that production code needs.

Task API

export type TaskStatus =
  | 'RECEIVED' | 'RUNNING' | 'RETRY_WAIT' | 'APPROVAL_WAIT' | 'EXECUTING'
  | 'EXEC_UNKNOWN' | 'VERIFYING' | 'COMPLETED' | 'ESCALATED' | 'FAILED' | 'CANCELLED';

export interface Principal { tenantId: string; subject: string }          // from the verified token, never the body

export interface TaskRepo {
  // One transaction: insert the task and an outbox row; on (tenant, idemKey) conflict return the existing task.
  insertOrGet(t: { tenantId: string; idemKey: string; payloadHash: string; input: unknown }):
    Promise<{ task: Task; created: boolean }>;
}

export async function createTask(
  p: Principal, idemKey: string, body: { message: string; channel: 'chat' | 'email' }, repo: TaskRepo,
): Promise<TaskView> {
  const payloadHash = sha256(canonicalJson(body));
  const { task, created } = await repo.insertOrGet({ tenantId: p.tenantId, idemKey, payloadHash, input: body });
  if (!created && task.payloadHash !== payloadHash) {
    throw new HttpError(422, 'idempotency_key_reused_with_different_payload');
  }
  return toView(task);                                                   // 202 + task id, status, trace id
}

Checkpoint store

export interface CheckpointStore {
  load(taskId: string): Promise<TaskState>;
  // Compare-and-swap: throws VersionConflict if another writer committed first.
  commit(taskId: string, expectedVersion: number, patch: StatePatch): Promise<TaskState>;
  acquireLease(taskId: string, workerId: string, ttlMs: number): Promise<Lease | null>;
  renewLease(lease: Lease): Promise<boolean>;
  release(lease: Lease): Promise<void>;
}

-- commit (one transaction); the state row and its event move together
UPDATE tasks SET status = $3, state = $4, version = version + 1, updated_at = now()
 WHERE id = $1 AND version = $2;                       -- 0 rows updated => VersionConflict
INSERT INTO task_events (task_id, seq, type, payload, trace_id) VALUES ($1, $2 + 1, $5, $6, $7);

-- acquireLease
UPDATE tasks SET lease_owner = $2, lease_expires_at = now() + interval '30 seconds'
 WHERE id = $1 AND status NOT IN ('APPROVAL_WAIT','RETRY_WAIT','COMPLETED','FAILED','CANCELLED','ESCALATED')
   AND (lease_owner IS NULL OR lease_expires_at < now())
RETURNING version;

Orchestrator

export async function runTask(taskId: string, workerId: string, d: Deps): Promise<void> {
  const lease = await d.store.acquireLease(taskId, workerId, 30_000);
  if (!lease) return;                                  // owned elsewhere, parked, or terminal
  try {
    let s = await d.store.load(taskId);
    while (isRunnable(s.status)) {
      const patch = await d.trace.span('step.' + s.status, { 'task.id': taskId }, () => step(s, d));
      s = await d.store.commit(taskId, s.version, patch);   // VersionConflict: we lost ownership, stop
      await d.store.renewLease(lease);
    }
  } finally {
    await d.store.release(lease);
  }
}

async function step(s: TaskState, d: Deps): Promise<StatePatch> {
  switch (s.status) {
    case 'RECEIVED':     return { status: 'RUNNING' };
    case 'RUNNING':      return decide(s, d);           // one model call: read, side-effect proposal, or final draft
    case 'EXECUTING':    return executePending(s, d);   // revalidate grant, write intent, call, classify
    case 'EXEC_UNKNOWN': return reconcile(s, d);        // look first; never blind-retry
    case 'VERIFYING':    return verify(s, d);           // deterministic checks, then reply
    default:             throw new Error('not runnable: ' + s.status);
  }
}

async function decide(s: TaskState, d: Deps): Promise<StatePatch> {
  const out = await d.models.propose(s.history, s.versions);        // schema-validated Proposal
  if (out.kind === 'final_draft') return { status: 'VERIFYING', draft: out.draft };
  const call = toToolCall(s, out);                                  // stable action hash + idempotency key
  const decision = d.policy.evaluate(call, s.facts, s.versions.policy);
  switch (decision.kind) {
    case 'deny':           return { observation: denied(call, decision.reason) };   // the model may re-plan
    case 'needs_approval': return { status: 'APPROVAL_WAIT', pendingAction: call,
                                    approval: await d.approvals.request(call, decision) };
    case 'allow':          return call.sideEffect === 'none'
                                    ? runRead(call, s, d)
                                    : { status: 'EXECUTING', pendingAction: call };
  }
}

Policy and approval gate

export type Decision =
  | { kind: 'allow' }
  | { kind: 'deny'; reason: string }
  | { kind: 'needs_approval'; reason: string; ttlMs: number };

export function evaluate(call: ToolCall, facts: Facts, policyVersion: string): Decision {
  if (call.tool !== 'refunds.create') return { kind: 'deny', reason: 'unknown_tool' };
  const pay = facts.payments[call.args.payment_id];                    // from the ledger, never from the model
  if (!pay || pay.status !== 'captured') return { kind: 'deny', reason: 'payment_not_captured' };
  if (!facts.duplicateCaptureConfirmed(pay.id)) return { kind: 'deny', reason: 'no_duplicate_capture' };
  if (call.args.amount_minor > pay.capturedMinor - pay.refundedMinor) return { kind: 'deny', reason: 'exceeds_refundable' };
  return call.args.amount_minor >= 150_000
    ? { kind: 'needs_approval', reason: 'POL-REF-004@4: >= INR 1,000', ttlMs: 30 * 60_000 }
    : { kind: 'allow' };
}

export interface ApprovalGrant {
  approvalId: string; actionHash: string; policyVersion: string;
  approverId: string; requesterId: string; expiresAt: Date; usedAt?: Date;
}

export function validateGrant(g: ApprovalGrant, call: ToolCall, currentPolicy: string, now: Date):
  { ok: true } | { ok: false; reason: 'used' | 'action_changed' | 'expired' | 'self_approval' | 'policy_changed' } {
  if (g.usedAt)                         return { ok: false, reason: 'used' };
  if (g.actionHash !== call.actionHash) return { ok: false, reason: 'action_changed' };
  if (now >= g.expiresAt)               return { ok: false, reason: 'expired' };
  if (g.approverId === g.requesterId)   return { ok: false, reason: 'self_approval' };
  if (g.policyVersion !== currentPolicy) return { ok: false, reason: 'policy_changed' };   // re-evaluate, maybe re-approve
  return { ok: true };
}

Tool executor

export type ToolOutcome =
  | { kind: 'ok'; result: unknown }
  | { kind: 'rejected'; code: string }                  // the service says it did not apply the request
  | { kind: 'unknown'; reason: 'timeout' | 'connection_error' };   // it may or may not have applied it

export async function executeTool(call: ToolCall, ctx: Ctx, d: ExecDeps): Promise<ToolOutcome> {
  const spec = d.registry.get(call.tool, ctx.tenantId);               // discovery is per tenant
  const v = spec.validate(call.args);
  if (!v.ok) return { kind: 'rejected', code: 'bad_args' };
  if (!spec.scopeAllows(ctx.stepScope)) return { kind: 'rejected', code: 'outside_scope' };

  const cred = await d.vault.mint(spec.name, ctx.tenantId, { ttlSeconds: 60 });
  try {
    const result = await withTimeout(spec.timeoutMs, (signal) =>
      d.adapters.invoke(spec, call.args, { idempotencyKey: call.idempotencyKey, credential: cred, signal }));
    return { kind: 'ok', result };
  } catch (e) {
    if (isDefinitiveRejection(e)) return { kind: 'rejected', code: errorCode(e) };
    return { kind: 'unknown', reason: e instanceof TimeoutError ? 'timeout' : 'connection_error' };
  } finally {
    await d.audit.record({ call, tenant: ctx.tenantId, traceId: ctx.traceId });
  }
}

Verifier and tracing

type Claim =
  | { type: 'duplicate_charge'; paymentIds: string[] }
  | { type: 'refund_submitted'; refundId: string; amountMinor: number; status: 'processing' | 'settled' }
  | { type: 'policy_cited'; doc: string; version: number };

export function verifyClaims(claims: Claim[], facts: VerifiedFacts): { ok: boolean; failures: string[] } {
  const failures: string[] = [];
  for (const c of claims) {
    if (c.type === 'duplicate_charge' &&
        (c.paymentIds.length < 2 || !c.paymentIds.every((id) => facts.captures.includes(id)))) {
      failures.push('duplicate_charge_unsupported');
    }
    if (c.type === 'refund_submitted') {
      const r = facts.refunds[c.refundId];
      if (!r || r.amountMinor !== c.amountMinor || r.status !== c.status) failures.push('refund_mismatch:' + c.refundId);
    }
    if (c.type === 'policy_cited' && !facts.policyDocs.some((p) => p.id === c.doc && p.version === c.version)) {
      failures.push('policy_not_retrieved:' + c.doc);
    }
  }
  return { ok: failures.length === 0, failures };
}

// Tracing wrapper using the OpenTelemetry JavaScript API (check the current API docs for your version).
import { trace, SpanStatusCode } from '@opentelemetry/api';
const tracer = trace.getTracer('agent-platform');

export function span<T>(name: string, attrs: Record<string, string | number | boolean>, fn: () => Promise<T>): Promise<T> {
  return tracer.startActiveSpan(name, { attributes: attrs }, async (s) => {
    try { return await fn(); }
    catch (e) { s.recordException(e as Error); s.setStatus({ code: SpanStatusCode.ERROR }); throw e; }
    finally { s.end(); }
  });
}

Everything above is now put under load by one incident. Task tsk_8472_a reached the policy gate with a proposal to refund ₹2,000 on pay_8472_b, the gate returned needs_approval (the amount is at or above ₹1,000 in POL-REF-004@4), and the task parked. A support lead granted approval apr_5521, bound to action hash sha256:a91c4e07…, nine minutes later at 10:11:17.199Z (illustrative timestamps throughout). What follows is the part the prototype could not survive.

Before the call: validation is the last thing before the side effect

The approval event wakes the task and a worker (w-07) leases it. It does not trust the event. It loads the pending action from the checkpoint and runs validateGrant: the grant is unused, its action hash equals the pending action's hash (so it cannot approve a different amount or payment), it has not expired (granted 540 s into a 1,800 s window), the approver differs from the requester, and the policy version is still POL-REF-004@4. Revalidation here is a 40 ms policy re-evaluation against the facts in the checkpoint; if a refund appeared in the meantime, the idempotency key, not a fresh read, is what protects the payment (a design that needs fresher facts can add a read step and pay for it in latency). Only then does it commit checkpoint 12, which stores the intent, the idempotency key refund:req-8472:pay_8472_b and the consumption of the grant in one transaction. This ordering is the heart of the design: the platform has written down that it is about to act before it acts.

The timeout: a fact about the network, not about the refund

The executor mints a 60 s credential, sends POST /refunds with the idempotency key, and waits. The payments service accepts the request and creates refund rf_3301, but the response is lost in transit, and the executor's 5 s timeout expires. What the executor knows is only that it got no answer. Classifying that as failure would invite a retry that, without the key, would create a second refund; classifying it as success would tell the customer something unverified. The honest outcome is unknown, so the orchestrator commits checkpoint 13 with status EXEC_UNKNOWN, and the status endpoint tells the customer "we are confirming your refund", not "refunded".

The crash: nothing was in memory that mattered

Moments after ckpt_13, worker w-07 is killed by an unrelated node reboot. Its lease (30 s) stops being renewed. Under the prototype this would have been the end: the intent, the unknown outcome and the idea of an approval were all stack variables. Here the lease expires at 10:11:47.259Z, the queue redelivers the task, and w-12 acquires it at 10:11:48.000Z, 25,741 ms after the checkpoint (the trace measures 25,751 ms from the end of the failed call). If the queue had also delivered it twice, the second delivery would have failed to take the lease. w-12 loads checkpoint 13 and sees EXEC_UNKNOWN, with the pending action and key.

Reconciliation: look before you act

reconcile does not call refunds.create. It calls the registry's reconcile tool,refunds.get_by_key, a read, and branches on the result. Found: the payments system returns refund rf_3301, ₹2,000 for pay_8472_b, status processing. The amount and payment equal the intent, so the effect is confirmed exactly once, and the task moves toVERIFYING. Not found: the call did not take effect, and a retry is permitted with the same key. Approval covers the intent, not the number of attempts, so the same grant still applies if it has not expired; after expiry the task would return to APPROVAL_WAIT for a fresh grant. Found but different, or the lookup itself failing repeatedly: the outcome cannot be determined, and the task is escalated to a human with the full trace instead of guessing.

Verification and the customer message

The model drafts the reply from verified facts, not from its memory of the conversation. The draft comes with its claims, and verifyClaims checks each against what the tools returned. The claim refund_submitted rf_3301, 200000, processing matches the reconcile observation, and the claim that two captures exist matches the ledger. A draft that said "your refund has been completed" would carry status: settled and fail, since the ledger says processing; the verifier would send it back for a redraft. The final reply (illustrative) reads:

"Hi, I checked order 8472. You were charged twice for ₹2,000, and I've submitted a refund of ₹2,000 for the
 duplicate charge (reference rf_3301). It has been accepted and is processing, so it hasn't reached your
 account yet. Under our refund policy (POL-REF-004, version 4) it should appear within 5 to 7 business days.
 I'll message you if anything changes."

{ "task_id": "tsk_8472_a", "version": 14, "status": "COMPLETED",
  "outcome": { "refund_id": "rf_3301", "amount_minor": 200000, "refund_status": "processing", "verified": true },
  "reconciled": { "unknown_at": "ckpt_13", "found_by": "idempotency_key", "second_refund_created": false } }

Count what carried the incident. Idempotency keys prevented a duplicate effect. Checkpoints preserved intent and the unknown outcome across a crash. Leases and CAS commits made recovery automatic and single-writer. A classified timeout stopped the blind retry. The approval grant, bound to a hash and consumed with the intent, made the recovery safe without asking a human twice. The verifier prevented an overclaim, and the trace let an engineer see all of it afterward. Remove any one and the same timeout produces a different story: a second ₹2,000, a stuck task, an unauthorized retry, or a customer told their money has arrived.

Trust is earned in steps, and each step is a decision made on evidence rather than on calendar dates. The order follows the cost of being wrong: first learn what the agent says, then what it may touch, then what it may do with a human watching, and only then what it may do alone.

Read the figure as a sequence of questions. Shadow asks whether the agent would have done the right thing on real traffic, at zero customer risk; its gate is the golden suite plus agreement with human outcomes. Read-only assistant exposes the agent's investigations and drafts to staff who send everything themselves: it measures factual accuracy, latency and cost against real cases, and it is already useful. Notice that this phase can be a synchronous service; the durable machinery is built for the next one. Bounded tools introduces the first side effects, deliberately low-risk ones, which is where idempotency keys and reconciliation are exercised for real before money is involved. Approved refunds adds the high-risk action with a human on every instance; the rejection rate is a direct measure of proposal quality, and approval latency is measured as a human-process SLO. Bounded autonomylets the agent execute only the narrow class where evidence is strongest, a small amount confirmed by the ledger (the policy would permit more; the rollout deliberately uses a smaller, lower-risk slice first), and expands traffic share only while online SLOs hold.

The gates are go/no-go, not targets to be met eventually. A failed gate holds the phase or rolls it back; it is not argued away. Two rules keep them honest. A single safety failure (an unauthorized action executed, a duplicate refund) fails the gate regardless of averages, because the cost of the tail dominates. And the evidence must come from the phase being exited: shadow results do not justify enabling writes. All figures in the diagram are illustrative; the real thresholds belong to the business owner of the risk, not to the engineering team alone.

A compact review list that maps to the chapters above. An agent that cannot answer each line is not ready to take real traffic.

With the machinery in hand, return to the question the earlier lessons left open: what shape of agent does this business case justify? The review asks what in the task is genuinely open-ended, and gives everything else to code.

The refund task has a stable backbone that is known before the customer writes: fetch the order and payments, check the policy, decide eligibility, request approval where required, execute with a key, verify, reply. The order is mandated by compliance, not discovered. What is open-ended is smaller: interpreting the customer's message, choosing which investigative reads matter when the situation is unusual (an authorization hold that looks like a double charge, a partial refund already issued), judging ambiguous evidence, and writing the reply. A free-roaming single-agent loop would make the mandated steps optional, because a model could skip verification or reorder the gate, and would spend a call to ask for what the workflow already knows it needs; the cost section showed one such call is worth ₹1.30 per task and 1.56 s (5.664 − 4.364 and 11,084 − 9,524 ms). A fully deterministic workflow cannot read the free-text complaint or handle unusual situations without a pile of special cases.

Choice: a deterministic workflow with bounded model decision points, which is a single agent operating inside steps the workflow defines. The state machine, gate, approval, execution, reconciliation and verification are code. The model has two jobs: propose the next read tools and the refund action within the step's allowed tool set, and draft the reply from verified facts. Constraints that go with the choice: at most 8 model calls per task; each step exposes only its own tools; a model cannot add steps, change scope or grant approval; and anything outside the golden set's coverage routes to a human.

Multi-agent is not justified here, and the reason can be stated. The task has no independent specialist domains with separate tool permissions and teams, no sub-problems that run usefully in parallel, and its context fits comfortably in one window (6,250 tokens at the end). Splitting it adds coordination cost: a minimal supervisor hop is one extra call with the same 3,200-token prefix and about 150 output tokens, which in the toy model is 3.2 × 0.20 + 0.15 × 0.80 = ₹0.76 and 150 + 320 + 1,200 = 1,670 ms before any specialist has done anything, plus handoff messages that can be lost or misinterpreted and a larger trace to read (see Multi-Agent Systems for what coordination adds and costs). The decision would change if the product grew into separate fraud, shipping-logistics and billing investigations owned by different teams with different credentials and data boundaries, or if contexts routinely exceeded one window: then a multi-agent or multi-workflow design pays for itself by isolating permissions and context, and the same platform (queue, checkpoints, gate, tracing) carries it unchanged.

The review itself is a repeatable artifact: restate the business case, list which steps are mandated versus open-ended, price the design's calls and latency with the toy model, name the controls each risk requires, and record the conditions under which the decision should be reopened. For the single-agent building blocks that sit inside each step, see Single-Agent Architecture.

Follow any arrow in the reference architecture far enough and it reaches something the agent layer does not control. The model gateway's concurrency cap of 40 came from a provider's limits or from the capacity of your own serving fleet. The per-call latency formula's prefill and decode terms are properties of how inference is served. The cost model's price per token is a statement about GPU utilization, batching and memory. The platform you have designed is only as good at its worst dependency, and those dependencies are infrastructure problems: where models run, how they are scaled and kept within latency targets, and how their cost is controlled.

That is where the track goes next. The agent platform answers "how do we run a reliable agent on top of a model"; AI infrastructure answers "what runs the model", including training and serving, compute and storage, and the operational layer that makes a gateway's concurrency cap a number you can plan around. If you have followed the refund agent from a definition through components, the loop, tools, planning, a single agent, multiple agents, MCP and guardrails to this platform, you can now trace a request across all of it. Next, go below it.