AI Infrastructure Overview

Follow a single question from a chat box all the way down through a typical AI serving stack: the application, the model, the compute cluster, and the serving and monitoring systems that make the answer possible. ChatGPT is only a familiar entry point: the stack shown is a generic, illustrative architecture, not a description of OpenAI's actual systems.

What is AI infrastructure? It is the compute, storage, networking, data pipelines, serving layer and observability that let a model be built, run and kept healthy. Beyond the model's mathematics, that means the accelerators that execute it, the memory and storage that hold its weights and data, the networks that move tensors and requests, the registry that versions model artifacts, the runtimes and endpoints that answer requests, and the security, monitoring and cost controls wrapped around all of it. Its architecture is organised around two lifecycles, an offline lane that produces a versioned model artifact (data, training, evaluation) and an online lane that turns that artifact into answers for live users (serving, inference, operations). The two lanes meet at one object, the model artifact in a registry.

Not every application operates all of it. A team that calls a hosted LLM API owns the application, its data and its policies, and rents everything below the API; a team that deploys its own model also owns the endpoint, runtime, accelerators and their failure modes. Training is done by far fewer organisations than inference. This chapter maps the whole AI infrastructure stack using one support question, then one training job, then the bytes and the milliseconds, so that the twelve specialist lessons that follow each have an obvious place to attach.

Running example for the whole track: an enterprise support assistant. A customer success agent at a tenant called acme types, "Can a customer with a duplicate charge get an automatic refund in the EU?"and presses Enter. Walk through what has to happen before the first word appears. The browser sends an authenticated HTTPS request to an API gateway, which validates the session token, rate-limits the tenant and forwards the call to the application's orchestrator. Everything up to here is ordinary web infrastructure and has nothing specific to AI.

The orchestrator is the application's own service, and it does the work that makes an answer grounded. It embeds the question, searches the tenant's document index with an authorisation filter so that only documents this user may read are candidates, and assembles a prompt from the system instructions, the retrieved passages and the question. If the question also asked about a specific order, the orchestrator would call an order-lookup tool under the user's permissions. A language model cannot see live transactions unless some authorised code fetches them and puts the result in the prompt; the model only ever receives text.

Only then does the prompt cross into model-serving territory: a request to a model endpoint, which may be a provider's API or your own service. That endpoint queues the request, an inference runtime tokenises the prompt, runs prefill (processing the whole prompt in parallel to produce the first token), then decode (generating the remaining tokens one step at a time) on an accelerator, and streams tokens back as they are produced. Every hop emits traces, token counts and costs to the observability system. The figure shows the same trip as a sequence; the dividing line between "application services" and "model-serving services" is the fifth lifeline, and moving that line is the main infrastructure decision this chapter discusses.

A definition by layers ("hardware, software, platform") hides the useful question: which concrete jobs does a team have to do for a model to serve traffic? There are ten, and they are best remembered by the failure each one prevents.

Compute executes the arithmetic. Training and large-batch work are throughput problems; interactive inference adds latency requirements. Memory decides what fits: weights, optimiser state, activations and the per-request attention cache all have to be resident in fast accelerator memory while the model runs, so a model that is too big simply does not start. Storage holds datasets, checkpoints and the artifact registry, and has to stream data fast enough to keep accelerators busy. Networking carries user requests, but in distributed training and multi-accelerator serving it also carries tensors between devices, which makes it part of the compute path rather than plumbing.

Data pipelines turn raw text and documents into training sets, evaluation sets and, for retrieval, indexed chunks; most model-quality failures trace back here. Model artifacts are the versioned bundle of weights, tokenizer and configuration with an identity you can pin and roll back. Inference runtimes are the software that loads those weights and executes them efficiently with batching and caching. Serving endpoints wrap runtimes with an API, health checks, routing, versioning and autoscaling. Security covers tenant isolation, secrets, what may be sent to which model, and what a tool may do. Monitoring and cost governance make quality, latency and spend visible, because token-priced or GPU-hour-priced systems can fail quietly by becoming expensive rather than by erroring.

Which of these a given team must own depends on how it consumes the model. A hosted-API consumer still owns data pipelines for retrieval, security of what it sends, application-level monitoring and cost tracking, and has no accelerators, no runtime and no registry of weights to worry about. A self-hosting team owns all ten. A team that fine-tunes an open model and serves it owns the offline lane as well. Reading the rest of this chapter, keep asking "which of these is mine?"; the hosted-versus-self-hosted section answers it concretely.

The infrastructure also does not include the model's knowledge. Weights encode patterns from training data and have a cutoff; what the assistant knows about this week's refund policy lives in your document index and arrives through the prompt. For why that matters see Why RAG Exists, and for the limits of what a prompt can hold see Context Window.

The most common conceptual error about AI infrastructure is to picture one pipeline in which data goes in one end and answers come out the other. There are two systems with different clocks, different owners and different failure costs. The offline lane runs for hours to weeks, is judged by model quality and cost per run, and tolerates being paused or restarted from a checkpoint. The online lane runs continuously, is judged by latency, availability and cost per request, and a minute of downtime is visible to users. They share hardware vendors and some libraries, but almost nothing operationally: a training cluster that is idle at night is not serving customers, and a serving fleet is rarely able to train.

The artifact is the handshake. What serving needs from training is small and precise: the weights in a stated numeric format, the tokenizer (a different tokenizer silently changes what the same bytes mean), the model configuration, and metadata saying which data, code and evaluation produced it. A registry entry makes that bundle immutable and addressable by version and content digest, so that "roll back to the previous model" is an operation rather than an archaeology project. Hosted-API consumers still have this object; it just lives in the provider's registry, and the version you pin in an API call, when the provider supports pinning, is your only handle on it.

Two details of this lifecycle matter for infrastructure. First, a checkpoint is not an artifact. A training checkpoint carries optimiser state needed to resume training and is several times larger than the weights needed for inference; the next section works through the numbers. Second, the canary state exists because offline evaluation cannot reproduce production traffic: real prompts are longer, noisier and more adversarial than the evaluation set, and the first place the version meets them is the small slice of traffic you chose to expose. Model Serving covers the rollout mechanics.

Here is a deliberately small, illustrative job, a full fine-tune of an 8-billion-parameter open model on support transcripts, so that every stage of the offline lane has a duration and a size. All numbers are assumptions for teaching, not measurements. The dataset is 50,000 example conversations averaging 600 tokens after tokenisation. That is 50,000 × 600 = 30 million tokens per pass over the data (an epoch), and three epochs process 90 million tokens. With a global batch of 64 sequences, each step consumes 64 × 600 = 38,400 tokens, an epoch takes ⌈50,000 / 64⌉ = 782 steps, and three epochs take 2,346 steps.

Suppose the cluster delivers 20,000 training tokens per second end to end; delivered, not peak, since data loading, communication and idle gaps all reduce it below any hardware datasheet. Then pure compute time is 90,000,000 / 20,000 = 4,500 s = 75 minutes, about 1.92 s per step (38,400 / 1.92 ≈ 20,000). The stages around training are not free: ingesting and cleaning the raw transcripts (removing duplicates, redacting personal data, filtering low-quality examples) is assumed to take 40 minutes, tokenising and sharding 15 minutes, and after training an evaluation run of 10 minutes and a registration step of 5 minutes.

The checkpoint arithmetic explains both the pause in the timeline and the difference between a checkpoint and an artifact. A common mixed-precision recipe with the Adam optimiser stores 2 bytes per parameter for 16-bit weights plus 12 bytes for optimiser and master state (a 32-bit master copy and two 32-bit moment estimates), about 14 bytes per parameter in total. For 8 × 10⁹ parameters that is 112 GB (decimal; 104.3 GiB) per checkpoint, of which only the weights, 16 GB (14.9 GiB), are needed to serve. Checkpoints are written every 500 steps, at steps 500, 1,000, 1,500 and 2,000. If a checkpoint is written synchronously to storage that sustains 2 GB/s, each one stalls training for 112 / 2 = 56 s, so four of them cost 224 s ≈ 3.7 minutes, the reason the training bar is 79 minutes rather than 75. (Real systems overlap writes with compute and shard them across workers; that trade-off is a training-lesson topic.) The artifact published to the registry is the 16 GB of weights plus tokenizer and configuration, not 112 GB.

Why does a job need checkpoints at all? Because at scale something fails during a run: a node, a network link, a data shard that cannot be read. Without a checkpoint, a failure at step 2,300 restarts at step 0; with them, it loses at most 500 steps, about 16 minutes at 1.92 s per step. That reasoning grows steeply with cluster size and run length, which is why distributed training has its own failure-handling machinery. For the mechanics of data, tensor and pipeline parallelism, gradient exchange and optimiser sharding see Training Infrastructure and Parallel Processing.

Inference serves one request at a time from the user's point of view, but the system is built around serving many at once. The reason is the shape of the work. Prefill processes all prompt tokens together, so it involves a great deal of arithmetic per byte of weights read and keeps the accelerator's compute units busy. Decode then generates one new token per step per request, and each step has to read essentially the whole set of model weights from accelerator memory to produce a single token for a single request. That is far less arithmetic per byte, so decode is typically limited by memory bandwidth, not compute. The remedy is batching: if sixteen requests are in decode simultaneously, one pass over the weights produces sixteen tokens, one per request, and the cost of reading the weights is shared.

This is why serving systems have a scheduler between the endpoint and the GPU. It admits requests into the running batch as slots free up (continuous batching), keeps per-request attention state (the KV cache, whose memory cost grows with prompt and output length) in accelerator memory, and decides who waits when memory is short. The details of that are the subject of Inference Infrastructure; what you need here is the vocabulary of performance, because three different numbers get called "throughput" and they are not interchangeable.

Token throughput is output tokens per second across the replica. Request throughput is completed requests per second, which depends on how many tokens each request produces. Per-stream speed is the tokens per second one user sees, the inverse of inter-token latency. Take the illustrative replica used in the timing trace below: a batch of 16 concurrent streams, each emitting a token every 20 ms (50 tokens/s per stream). Token throughput is 16 × 50 = 800 tokens/s. If requests average 200 output tokens, the decode-side ceiling is 800 / 200 = 4 requests/s. A request lasts about 4.33 s, so Little's law (average number in the system = arrival rate × time in system) gives 4 × 4.33 ≈ 17.3 concurrent requests, more than the 16 slots, so the slots, not the GPU's arithmetic, are the binding limit: 16 / 4.33 ≈ 3.7 requests/s. And that still ignores prefill, which competes for the same GPU. A burst of long prompts can slow every active stream's token cadence. Peak throughput numbers measured with ideal batches therefore tell you little about the p95 latency a customer actually sees; the capacity of the service is what you can sustain while still meeting the latency objective, found by load testing with realistic prompt and output lengths.

The figure below recomputes that example. At 16 streams and 200 output tokens it reproduces the 4 requests/s ceiling and 3.7 requests/s slot-bound rate; change either control to see which limit binds first.

A single mental picture clears up most confusion about "where the model runs": a hierarchy of places that hold bytes, each faster and smaller than the one before it. Model weights begin in object storage, a durable network-attached store. To serve them, a replica copies them across the network to a local NVMe disk or directly into host RAM, then across the host's PCIe bus into the accelerator's high-bandwidth memory (HBM, also called VRAM for GPUs), where the compute cores can use them. The CPU is not idle in this picture; it runs the orchestration, the tokeniser, the scheduler and the code that launches accelerator kernels. The accelerator does the dense arithmetic. Multi-accelerator deployments add a fourth path: a fast interconnect between devices that carries partial results when one model is split across several of them.

Put numbers on it with the same 8-billion-parameter model in 16-bit precision: 8 × 10⁹ parameters × 2 bytes = 16 GB in decimal units, 14.9 GiB in binary. Loading from remote storage over an effective 10 Gbit/s (1.25 GB/s) link takes 16 / 1.25 = 12.8 s; from a local NVMe disk at an assumed 3 GB/s, 5.3 s; and the final hop across PCIe at an assumed 25 GB/s is 0.64 s. These are the cold-start costs that decide how quickly an autoscaler can add a replica: if the weights are not already on local disk, a new replica is not useful for at least the network-limited 12.8 s plus runtime start-up, which is why a spike can overwhelm a service before new capacity arrives, and why model-serving systems keep weights close to the node.

Now consider what happens per request. The prompt is text, a few kilobytes: 3,000 tokens as 4-byte integer IDs is 12 KB. That is negligible next to 16 GB. Weights do not travel with each request; they already sit in HBM, and the request moves to the weights. The expensive movement is inside the GPU: each decode step reads the full weights from HBM. At an assumed 2,000 GB/s, 16 GB / 2,000 GB/s = 8 ms is a lower bound on one decode step, regardless of how fast the arithmetic is. The 20 ms per step used in this chapter's trace is above that bound because real steps also read KV-cache data, launch many small kernels, run sampling and scheduling, and never reach peak bandwidth. The gap between a datasheet bound and an observed number is the room where runtime engineering lives.

Prefill is the reverse case. Counting roughly 2 floating-point operations per parameter per token for a dense transformer's forward pass (ignoring the attention term), a 3,000-token prompt costs about 2 × 8 × 10⁹× 3,000 = 4.8 × 10¹³ operations. If prefill takes 200 ms, the delivered rate is 2.4 × 10¹⁴operations per second, 240 TFLOP/s. Whether that is plausible depends on the accelerator; what matters is that it should be compared with measured delivered throughput on your stack, never with a theoretical peak, because kernel efficiency, memory stalls and precision all push delivered below peak. Prefill is bound by compute and decode by memory bandwidth, and that one fact explains why a single GPU can look underused during decode while users still wait: arithmetic units idle while HBM is saturated.

Which bottleneck binds also depends on scale. Within one accelerator, memory capacity (does the model plus its caches fit?) and bandwidth dominate; across accelerators, the interconnect starts to matter; across machines and regions, network latency and bandwidth do. A model too large for one device must be split, and every split turns some memory traffic into network traffic. Accelerator internals, their memory hierarchies and the choice between them are the subject of CPUs vs GPUs vs TPUs and AI Accelerators.

When you call a provider's LLM API, the physical infrastructure above does not disappear; it becomes someone else's. Their data centres hold accelerators, their schedulers batch your request with other customers', their registry holds the model version you named. The abstraction removes operational work, not physics: your request still waits in a queue, still goes through prefill and decode, and latency still varies with load. What the abstraction changes is which of the failures and costs are yours to handle.

Read the figure down the columns. In both, the top three rows are yours: the application that builds prompts, retrieval with tenant-level authorisation, and the policy for quotas, retries and fallbacks. These are never outsourced by buying an API. Moving from hosted to self-hosted turns the next five rows green: you now run the endpoint, pin and verify the weights, tune the runtime, obtain accelerator capacity and instrument the model tier. The two bottom rows stay with a cloud or colocation provider unless you run your own data centre.

The security and data consequences are not symmetrical. With a hosted API, prompts containing customer policies and, in the order-lookup case, customer data leave your network, so contract terms, data retention settings and regional processing options are part of the design. Self-hosting inside your network keeps those bytes in your boundary but adds exposure of its own: model weights are an asset to protect, the inference endpoint is an attack surface, and you are responsible for patching the serving stack. AI Security goes further.

Cost behaves differently too, and an illustrative calculation shows how. Take the support assistant at 100,000 requests per day with 3,000 input and 200 output tokens each. Assume a fictional API price of $1 per million input tokens and $4 per million output tokens. One request costs 3,000 × $1 / 10⁶ + 200 × $4 / 10⁶ = $0.003 + $0.0008 = $0.0038, or $380 per day. Assume instead two self-hosted GPU replicas (two for redundancy) at a fictional $2.50 per GPU-hour, always on: 2 × $2.50 × 24 = $120 per day, or $0.0012 per request at 100,000 requests/day. At the replica's modelled 3.7 requests/s, two replicas can serve about 7.4 requests/s, and the average load of 100,000 / 86,400 = 1.16 requests/s is only 16% of that; if the daily peak is three times the average, 3.5 requests/s is 47%.

Read that carefully before concluding self-hosting wins. The $120 excludes engineers who operate the fleet, on-call, evaluation of the model against a hosted frontier model, the cost of idle headroom for failures and spikes, storage, networking and the quality gap if the self-hosted model is less capable. The hosted figure scales to zero when traffic does; the fleet does not. Break-even depends on sustained utilisation, quality requirements, and how much the team values control, which is why the evolution path below starts with the hosted option and moves only when measurement says so. Cost Optimization treats this decision properly.

Now put timestamps on the sequence from earlier. Every number below is an illustrative assumption, measured from the moment the request reaches the gateway, so the user's network round trip (perhaps another 30 to 80 ms) is excluded. The model is self-hosted on the replica described above. The prompt is 3,000 tokens: system instructions, four retrieved passages and the question. The answer is 200 tokens.

TTFT (time to first token) is the sum of the stages that must finish before any answer is visible: 25 (gateway) + 20 (embedding) + 40 (search) + 10 (assembly) + 40 (network and queue) + 5 (tokenise) + 200 (prefill) + 10 (delivery of the first token) = 350 ms. Retrieval, at 60 ms, is 17% of it; prefill is 57%, and prefill scales with prompt length, so retrieving six passages instead of four (say, a 4,500-token prompt) would raise it by about 100 ms at the same implied rate of 15,000 prompt tokens/s (3,000 / 0.2 s). Queue wait is 11% and is the term that explodes under load: a replica running at its limit turns 40 ms into seconds.

Total latency adds decode. The first token is produced at the end of prefill, at 340 ms. The remaining 199 tokens each take a 20 ms step, 199 × 20 = 3,980 ms, so the last token is sampled at 340 + 3,980 = 4,320 ms and delivered at 4,330 ms. The visible streaming rate after the first token is 1 / 20 ms = 50 tokens/s, faster than most people read, so streaming makes the answer feel responsive even though completion takes 4.3 s. This is also why the two numbers need separate objectives. A service level objective on TTFT protects against queueing and long prompts; one on inter-token latency protects against batch saturation; a total-latency objective needs an assumption about output length, since doubling the answer to 400 tokens makes total latency 340 + 399 × 20 + 10 = 8,330 ms with an unchanged TTFT.

The same trace with a hosted API replaces the segments from "network + queue wait" through "prefill" with a single remote call whose internals you cannot see. You still measure TTFT and total latency at your own edge, and you still own the retrieval stage, which an API provider's status page never explains. Percentiles matter more than the average here: the trace is a single request, the p50 of a healthy service. If one request in twenty hits a cold cache, a long prompt or a full queue, the p95 can be several times the trace's figures, and users judge the service by the slow ones.

The multi-tier path above has a long series of dependencies, and reliability comes from deciding in advance what each failure means to the user. A principle that avoids most bad designs: degrade by being less capable and honest about it, never by being silently wrong. For a policy assistant, an answer produced without retrieved policy text is worse than no answer, because the model will still produce confident, fluent text, as described in Hallucinations.

Retrieval fails. The vector index times out after 2 seconds. The orchestrator should not send the question to the model as if nothing had happened. It can retry once within a budget (the TTFT budget is shared, so a retry that costs 2 s may breach the objective), then either answer with an explicit "I could not check the policy documents" or refuse and offer a support ticket. If the assistant has a non-policy mode, such as drafting a customer email, that mode may continue without retrieval. The order tool failing is milder: answer the policy half and say that live order status is currently unavailable. The model must be told, in the prompt, that the tool failed; otherwise it will fill the gap with plausible text. Tools and Tool Calling covers the contract.

The model service is overloaded. Requests accumulate in the queue; each added request lengthens the wait of every one behind it, so latency climbs until timeouts start, at which point clients retry and make it worse. Reliable systems put a bound on queue depth, reject early with a retry hint rather than accept work they cannot finish within the objective, and optionally route to a fallback: a smaller model, or a different provider, with the quality difference known and measured in advance. A fallback you have never evaluated is a gamble, not a plan.

A GPU replica dies. Capacity drops immediately, requests in flight on that replica are lost and the load balancer has to stop sending it new ones. What the user sees depends on timing. If no tokens have been streamed, the gateway can retry transparently on another replica, costing the prefill again. If tokens are already on screen, a transparent retry would repeat or contradict text the user has read, so the stream usually ends with an error event and the client offers to resume or retry. Replicas also die during start-up: a replica that is still loading its 16 GB for 12.8 s must not receive traffic, which is what readiness checks, as opposed to liveness checks, are for.

A provider or dependency is unavailable. This is the hosted-API analogue of a dead replica, with less visibility. Short timeouts, a circuit breaker that stops calling a failing dependency for a cooldown period, retries with backoff and jitter capped to a small number, and an explicit decision about whether a second provider is worth its integration and evaluation cost. Retries amplify: with three tiers each making three attempts, one user request can become 27 backend calls during an outage; budget retries at the edge and propagate a deadline instead.

Reliability is a user-visible contract as much as an engineering one. For the support assistant, the contract is: never present an ungrounded policy answer as grounded; always say what could not be checked; and never charge or modify an order as a side effect of a retried request. Write the contract down for each failure before building the mechanisms, then test it by injecting each failure. The scaling and observability lessons show how the same signals drive capacity decisions and diagnosis.

A useful way to lay out the whole system is to separate what moves per request from what changes the system's configuration. The data plane carries user work: gateway, orchestrator, retrieval, the model replicas and the tokens they produce. It has to be fast and highly available. The control plane decides what the data plane looks like: which model version is loaded where, how many replicas exist, which quotas and routes apply, when the document index is rebuilt. It can be slower and can briefly fail without dropping requests, provided that the data plane keeps running with its last known configuration. This distinction is what makes a rollout safe: the registry tells the rollout controller to load version v7 on two replicas, health checks pass, the gateway's route weights shift, and no request ever needed to talk to the registry.

Three placement ideas follow from the byte-movement picture. Data locality: the prompt carries retrieved text and sometimes personal data, so the index, orchestrator and model should sit close together, both for latency and to avoid sending regulated data across regions. Each cross-region round trip costs tens of milliseconds; if the orchestrator in one region made two sequential calls (embedding and search) to a service in another with an assumed 70 ms round trip, that adds 140 ms to the 350 ms TTFT before any other change, a 40% increase. Model versioning: pin the version in every request path, log it in every trace, and treat prompts, indexes and tool schemas as versioned artifacts as well, because an answer can change when any of the four changes. Regional placement of GPU capacity is also a supply question, not only a latency one: accelerators may not be equally available in every region, and regional autoscaling can be limited by quota.

Where each objective is measured determines what it can detect. TTFT and total latency are measured at the edge the user sees, the gateway or client, since anything measured only inside the model server omits retrieval, queueing and the network. Queue wait, prefill time and inter-token latency are measured inside the serving tier to diagnose which part of the edge number moved. Retrieval latency and tool latency are measured by the orchestrator. Quality (groundedness, citation correctness, task success) cannot be read from latency metrics at all and needs sampled evaluation. Availability objectives are judged by the outcomes users experience, including honest degraded answers, which you have to decide whether to count as success. AI Observability works through this.

The architecture diagrams so far show an end state that few teams need on day one. A better practice is to add each piece when a measurement justifies it. A typical growth path for the support assistant looks like the five steps below, and each step is a deliberate answer to a symptom rather than a milestone to reach.

1. 1

   Prototype: a hosted LLM behind one endpoint

   One service calls a provider API with a system prompt and the user's question. No GPUs, no registry, no retrieval. Add the cheap basics now: timeouts, a per-tenant rate limit, logging of token counts and latency, and a recorded set of 30 to 100 representative questions to compare later changes against. Symptom that ends this stage: answers about your own policies are wrong or outdated.
2. 2

   Add retrieval with authorisation

   Index policy documents with tenant and role metadata, filter by it at query time, and inject the passages into the prompt. Quality improves because the model now reads the policy; a new infrastructure problem appears, which is that the index is production data with its own freshness, access control and failure modes. See the retrieval pipeline and context injection.
3. 3

   Add an authorised tool for live order data

   The orchestrator exposes an order-lookup function that runs with the signed-in user's permissions and returns a bounded, validated result. It adds a second dependency to the reliability budget and a second place for tenant-isolation bugs. Not every application needs it, and it does not make the assistant an autonomous agent; it is one function call under application control.
4. 4

   Add caching, load balancing and observability when load demands

   When load tests or bills show repeated work, add layers deliberately: caching with tenant-scoped keys, and traces that tie each answer to a model, prompt and index version. Quotas per tenant protect the multi-tenant service from one customer's batch job.
5. 5

   Self-host a model only when the numbers justify the work

   Reasons that justify it: data cannot leave your boundary, sustained utilisation makes dedicated capacity cheaper than tokens, you need a fine-tuned or specialised model, or a latency or regional requirement a provider cannot meet. Then the offline artifact, the runtime, the fleet and the failure modes above become yours. See Model Serving and Production AI Architecture.

One design choice makes the later steps cheaper: keep the model behind a narrow interface, so the orchestrator does not know whether tokens come from a provider or from your cluster. The sketch below is illustrative pseudocode in TypeScript, not a specific provider's SDK; real request shapes and streaming protocols differ by provider and must be taken from their documentation.

interface CompletionRequest {
  tenantId: string;
  model: string;           // a pinned model version, logged with every trace
  prompt: string;          // system text + retrieved passages + question
  maxOutputTokens: number; // bounds decode time and cost
  deadlineMs: number;      // remaining time budget, propagated from the edge
}

interface TokenChunk { text: string; done: boolean; }

interface ModelClient {
  stream(req: CompletionRequest, signal: AbortSignal): AsyncIterable<TokenChunk>;
}

// Composition: try the primary, fall back only before any token was emitted.
class FallbackClient implements ModelClient {
  constructor(private primary: ModelClient, private backup: ModelClient) {}

  async *stream(req: CompletionRequest, signal: AbortSignal): AsyncIterable<TokenChunk> {
    let emitted = false;
    try {
      for await (const chunk of this.primary.stream(req, signal)) {
        emitted = true;
        yield chunk;
      }
    } catch (err) {
      if (emitted) throw err; // the user already saw text: surface the error, do not splice
      yield* this.backup.stream(req, signal);
    }
  }
}

Note what the code encodes: the deadline travels with the request so every tier knows the remaining budget; output length is capped because it is the largest driver of both latency and cost; and fallback happens only before the first token, for the reason given in the reliability section.

This chapter treated the accelerator as a box that holds weights and does arithmetic, and the stack around it as a set of jobs. Each remaining lesson takes one of those jobs and asks the next engineering question, and they group naturally by the problem you are trying to solve.

• "Why GPUs, and what is a TPU?" CPUs vs GPUs vs TPUs compares processor architectures and workload fit; AI Accelerators opens the device: memory hierarchy, interconnect and precision.
• "The model does not fit, or training is too slow." Parallel Processing explains how work is split and what communication costs; Training Infrastructure applies it to data, optimiser state and checkpoints.
• "Answers are slow or the GPU is idle." Inference Infrastructure covers prefill, decode, batching and KV memory; Model Serving covers endpoints, routing, rollout and health.
• "Traffic is growing." AI Infrastructure Scaling turns requests and tokens into capacity; Caching in AI Systems asks which repeated work is safe to avoid.
• "Something is wrong and I cannot tell what, or what it costs, or who can see it." AI Observability, Cost Optimization and AI Security take those three questions in turn.
• "Put it all together." Production AI Architecture integrates every layer into one deployable reference system.

The natural next step is hardware: decode is limited by memory bandwidth and prefill by compute, so the processor's design decides what the stack above it can achieve. The next lesson compares the architectures that execute these workloads and why GPUs became the default.

1. 1

   AI infrastructure is the compute, data, networking, serving and observability that surround a model; a team that calls a hosted API still owns the application, its data path, and its latency, cost and security boundaries.

2. 2

   Two lifecycles meet at the model artifact: offline (data to training to checkpoint to artifact) and online (queue, prefill, decode). A checkpoint is not a deployable artifact until serving has what it needs in a stable format.

3. 3

   When people say the system is slow they often mean different tiers (retrieval, tools, queue, prefill, decode), so measure each stage separately before blaming the GPU.

4. 4

   Define TTFT as the sum of everything before the first token and total latency as TTFT plus decode; token throughput and request throughput differ, so a service with long answers needs a very different fleet at the same request rate.

5. 5

   Decide in advance what happens when each dependency fails (retrieval timeout, overloaded model service, dead replica, provider outage), and carry the deadline with the request so each tier knows its remaining budget.

6. 6

   Start with the smallest working system behind a narrow model interface and add machinery only when a measurement demands it; compare self-hosting against a hosted API on total cost including the engineers who would run it.