AI Accelerators

Watch a CPU buckle under billions of calculations, a GPU step in with thousands of parallel cores, and then meet the wider family of chips — TPUs, NPUs, and custom silicon — that companies are racing to build for AI.

An AI accelerator is a processor, or a processor plus its attached memory and links, that is built so that the dense tensor arithmetic and the heavy data movement of neural networks cost less time and energy than they do on a general-purpose CPU. Where a CPU spends its silicon on a few flexible cores and large caches, an accelerator spends it on matrix units and high-bandwidth memory (HBM) aimed at training and large-model inference. GPUs, TPUs, NPUs and custom inference or training ASICs are all AI accelerator chips; they differ in how programmable they are, where they are deployed, and which trade-offs they make between peak arithmetic, memory capacity, memory bandwidth, power and software maturity. What decides whether an accelerator is a good choice is rarely its headline arithmetic rate. It is whether your model and its working memory fit, whether the data can be fed to the arithmetic units fast enough, whether devices can talk to each other fast enough, and whether your software stack can reach the hardware without heroic effort.

This lesson opens the device up. It follows bytes from host memory through high-bandwidth memory (HBM) and on-chip SRAM into matrix units, does the memory arithmetic that decides what fits, derives why single-stream text generation is usually limited by memory bandwidth, and then turns to precision, interconnects, power, software stacks and selection. It ends with a diagnosis of the most common production failure: a model that should fit and still runs out of memory. For the broader processor-architecture comparison see CPUs vs GPUs vs TPUs; for how work is split across devices see Parallel Processing.

Suppose the enterprise support assistant has outgrown a hosted API and the team wants to self-host a 7-billion-parameter model so that policy questions never leave their network. A first benchmark on a borrowed GPU produces two surprises. The model loads fine, yet a single user sees only a few dozen tokens per second. And when a load test raises concurrency, it does not slow down smoothly: at some point requests start failing with out-of-memory errors even though the weights alone occupy barely more than half of the device. These are two different bottlenecks, and they explain why accelerators look the way they do.

The first is arithmetic versus data movement. A language model layer is mostly a matrix multiplication: an activation matrix times a weight matrix. Multiplying two large matrices performs order n³ arithmetic on order n² numbers, so with enough reuse the arithmetic units, not the memory, are the limit. A general-purpose CPU has a few dozen wide cores and a memory system tuned for irregular, latency-sensitive code, so it cannot supply enough parallel multiply-accumulate lanes to exploit that reuse. An accelerator puts thousands of such lanes, packaged as matrix units, next to a very wide memory system. But when a request generates one token at a time, there is almost no reuse: every weight is fetched from memory, used for a single multiply-accumulate per sequence, and thrown away. The arithmetic units idle waiting for bytes. Fast silicon does not help when the problem is that the data is not there yet.

The second is capacity versus headroom. Weights are only one of several things that must be resident in device memory while a request is served. The attention key-value cache grows with every token of every active conversation, the runtime keeps scratch buffers, and the allocator wastes some space to fragmentation. “The model fits” is therefore a statement about the weights; whether the service fits depends on concurrency and context length too.

Specialised silicon exists because both problems are structural. Matrix units, wide memory stacks placed next to the compute die, a software-managed on-chip scratchpad, and high-speed links between devices are all responses to one of those two problems. Seeing which block answers which problem is the fastest way to read any accelerator data sheet.

The diagram below is a generic accelerator, not any vendor's floor plan. Real devices differ in how many of each block they have and what the blocks are called, but nearly all of them contain the same roles.

Matrix (tensor) units are fixed-function arrays that take small tiles of two matrices and perform every multiply and accumulate of the tile product in a handful of clock cycles. That fixed structure is the main reason an accelerator achieves far more arithmetic per watt than a CPU executing the same maths as scalar or short-vector instructions. Vector units handle everything that is not a dense matrix product: normalisation, activation functions, the softmax inside attention, elementwise adds, type conversions. They are far less numerous than the matrix lanes, so a model with a lot of non-matmul work can be limited here rather than in the matrix units.

HBM (high-bandwidth memory, called VRAM on many GPUs) is a stack of DRAM dies placed in the same package as the compute die. It trades capacity for width: many parallel channels give far higher bandwidth than the DIMM-based DRAM that a host CPU uses, but total capacity is limited by what can be stacked and packaged. Weights, activations, optimiser state and the KV cache all live here. The memory controllers schedule thousands of outstanding requests across channels; accesses that are contiguous and aligned use bandwidth well, scattered ones do not.

On-chip SRAM (register files, shared memory or scratchpad, and cache levels) is orders of magnitude smaller than HBM and much faster and cheaper in energy per byte. Its job is to hold the tiles currently being reused so that each byte fetched from HBM is used many times. Whether this memory is a hardware-managed cache or a software-managed scratchpad is one of the real architectural differences between families, and it is why kernel authors spend so much effort on tiling. DMA engines copy data between memory spaces without occupying the arithmetic units, which allows the next tile to be loaded while the current one is being computed. The host interface connects the device to the CPU and host DRAM for loading weights, sending inputs and returning outputs; it is far narrower than HBM, which is why you want the weights to cross it once, at start-up, and never again. Device links connect accelerators to each other; we return to them below.

Following one matrix multiplication through the hardware

The sequence below traces a single layer's weight matrix multiplication on one device. Nothing in it is specific to one vendor; the names of the steps vary.

Steps 2 to 6 repeat for every tile along the shared dimension; the accumulator is held on chip across them, so the large intermediate sums never visit HBM. That is the whole trick of tiling: bytes from HBM are reused across many arithmetic operations. The reuse is high when the activation matrix has many rows (a large batch of tokens, as in training or in processing a long prompt) and almost absent when it has one row (generating one token for one user). Keep that in mind: the next sections quantify it.

Every level of memory is bigger and slower than the one above it, and cheaper per byte. Performance engineering on an accelerator consists mostly of arranging for the data needed next to already be in the tier closest to the arithmetic.

The capacity labels are orders of magnitude, not specifications: they are meant to convey that each step down costs roughly one to two orders of magnitude in capacity versus speed. Two consequences follow for service design. First, anything that spills from HBM to host DRAM crosses the narrow host link, so offloading tensors is a last resort for latency-sensitive serving even if it works functionally. Second, loading a model is a storage and network event before it is a GPU event: a 14 GB set of weights must travel from object storage to host memory and across the host link before the first request can be served, which is why cold starts are measured in tens of seconds to minutes and why serving systems keep warm replicas.

What occupies device memory in each regime

Five kinds of data compete for HBM: model weights, activations (the intermediate tensors of a forward pass), gradients, optimiser state, and the KV cache, plus temporary workspaces and runtime overhead. Which ones dominate depends entirely on what you are doing.

Training must keep weights, gradients, optimiser state and the activations needed for the backward pass. Optimiser state is the surprise for newcomers: the Adam optimiser stores two extra numbers per parameter, and mixed-precision recipes usually also keep a full-precision master copy of the weights. The arithmetic below makes this concrete. Inference needs only weights, a small activation workspace, and the KV cache; there are no gradients or optimiser state. At low concurrency weights dominate; at high concurrency or long context the KV cache grows and can exceed the weights. Edge devices run inference with much smaller budgets and rely heavily on quantised weights.

The starting calculation is just a multiplication. A dense model with P parameters stored at b bytes per parameter needs P × b bytes for its raw weights. For an illustrative 7-billion-parameter dense model:

P = 7,000,000,000 parameters

2 bytes/param (BF16 or FP16):  7e9 x 2 = 14e9 bytes = 14 GB   (13.04 GiB)
1 byte/param  (FP8 or INT8):   7e9 x 1 =  7e9 bytes =  7 GB   ( 6.52 GiB)
0.5 byte/param (4-bit):        7e9 x 0.5 = 3.5 GB   before scales and metadata

Real quantisation formats vary. A 4-bit scheme typically stores a scale (and sometimes a zero point) per small group of weights, say one 16-bit scale per 128 weights. That adds 2 / 128 = 0.0156 bytes per parameter, so a nominal 4-bit format costs about 0.5156 bytes per parameter, or 3.61 GB for 7 billion parameters, not 3.5 GB. Some layers (embeddings, the output head, normalisation parameters) are often kept at higher precision, and some formats keep a few outlier channels unquantised. Treat “7 GB at one byte” as a lower bound on what you will observe, never a prediction of the allocator's total.

Training the same model

A common mixed-precision training recipe with the Adam optimiser stores, per parameter: a 2-byte BF16 working weight, a 2-byte BF16 gradient, a 4-byte FP32 master weight, and two 4-byte FP32 optimiser moments. Under those assumptions:

BF16 weights          2 bytes
BF16 gradients        2 bytes
FP32 master weights   4 bytes
Adam first moment (m) 4 bytes
Adam second moment (v)4 bytes
                     -------
                     16 bytes per parameter

7e9 x 16 = 112e9 bytes = 112 GB   (excluding activations, workspaces, fragmentation)

The consequence is blunt: this 7B model, which serves comfortably from 14 GB, needs about eight times that for its training state before a single activation is stored. A device with less than 112 GB of HBM cannot train it in this configuration on its own; training systems shard weights, gradients and optimiser state across devices (see Training Infrastructure) or reduce the optimiser footprint. Activations then add a term that scales with batch size and sequence length and can be traded for recomputation.

The second and third bars use the KV-cache figure that the next section derives. Notice what the quantised serving row shows: halving the weights saves 7 GB, but the KV cache at this concurrency is more than twice that size. Once concurrency is high, weight quantisation alone stops being the lever that matters.

During autoregressive generation, every new token's attention needs the keys and values of all earlier tokens in the sequence. Recomputing them at every step would be wasteful, so the engine caches them in HBM as it goes. For each token and each layer, the cache holds one key vector and one value vector per KV head. The size of the cache is therefore a straightforward product:

bytes_per_token = 2 x layers x kv_heads x head_dim x bytes_per_element
                  ^ one K and one V

cache_bytes     = bytes_per_token x tokens_per_sequence x concurrent_sequences

Take an illustrative architecture: 32 layers, 8 KV heads (grouped-query attention, where several query heads share one KV head), head dimension 128, stored in BF16 (2 bytes). These are plausible values for a model of this size class, chosen for the example rather than taken from a specific model card.

bytes_per_token = 2 x 32 x 8 x 128 x 2 = 131,072 bytes = 128 KiB (~0.131 MB)

one sequence, 4096 tokens:
  131,072 x 4,096 = 536,870,912 bytes ~ 0.537 GB

32 concurrent sequences, 4096 tokens each:
  0.537 GB x 32 = 17.18 GB

same model WITHOUT grouped-query attention (32 KV heads, 4x more):
  bytes_per_token = 524,288;   32 x 4096-token sequences = 68.7 GB

The 17.18 GB is larger than the 14 GB of weights, and it belongs to traffic, not to the model. This is the bridge between hardware and the application. Every policy passage that the retrieval step places in the prompt (see Context Injection and the context window lesson) consumes KV memory for the entire life of the request. Trimming a retrieved-context budget from 3,000 tokens to 1,500 is a hardware decision as much as a prompt-design decision. The scheduling consequences, such as paged KV allocation, prefix reuse and admission control, are the subject of Inference Infrastructure; this page only needs the size, because it determines which device you can buy.

Whether a kernel is limited by arithmetic or by memory is captured by one ratio, its arithmetic intensity: floating-point operations performed per byte moved from memory. A device has a matching ratio: its peak arithmetic rate divided by its memory bandwidth. This ratio is the “ridge point” of the roofline model. A kernel with intensity below the ridge point is bandwidth-bound: its time is bytes divided by bandwidth, and extra arithmetic capacity is wasted. A kernel above it is compute-bound: its time is operations divided by delivered arithmetic rate.

Now apply this to decoding. Generating one token for one sequence multiplies a vector by every weight matrix. Each weight is used for a single multiply-accumulate, which is two floating-point operations, but the weight is 2 bytes in BF16. The intensity is about 1 operation per byte. Compare that with a hypothetical device that sustains 200 TFLOP/s of dense BF16 arithmetic and has 1 TB/s of HBM bandwidth: the ridge point is 200 × 10¹² / 10¹² = 200 operations per byte. Single-stream decode sits at roughly 1 operation per byte, two orders of magnitude to the left of the ridge. The matrix units are almost entirely idle.

A toy decode estimate (illustrative, with its assumptions)

If every token requires reading every weight once and nothing else mattered, the upper bound on single-stream speed is bandwidth divided by weight bytes. The numbers below use a hypothetical device with 1,000 GB/s of memory bandwidth and the 7B model from above.

Assumptions: 1,000 GB/s HBM bandwidth (hypothetical), batch size 1,
             every weight read once per token, perfect bandwidth use,
             no cache reuse, no kernel overhead, no multi-device parallelism.

BF16 weights:   14 GB / 1,000 GB/s = 14.0 ms per token  ->  71.4 tokens/s ceiling
1-byte weights:  7 GB / 1,000 GB/s =  7.0 ms per token  -> 142.9 tokens/s ceiling

Add the KV cache read for one 4096-token sequence (0.537 GB):
  (14 + 0.537) GB / 1,000 GB/s = 14.54 ms per token -> 68.8 tokens/s

Arithmetic check: 2 x 7e9 = 14 GFLOP per token; at 200 TFLOP/s that is 0.07 ms.
  Memory time (14 ms) is ~200x the arithmetic time: bandwidth-bound.

The estimate teaches three things. Per-token latency for one user is set by bytes of weights divided by bandwidth, so a lower-precision format that halves the bytes can nearly double single-stream speed even though it does not change the arithmetic; this is why weight-only quantisation is popular for serving. A device with more arithmetic but the same bandwidth does not help this regime. And adding KV reads makes long contexts slower per token, because every generated token must also stream the whole cache.

Why batching changes the picture

The weight read is shared. If 8 sequences decode in the same step, the engine reads the weights once and applies them to eight activation vectors, while each sequence still reads its own KV cache. Under the same toy assumptions:

bytes per step = 14 GB weights (read once) + batch x 0.537 GB KV
step time      = bytes per step / 1,000 GB/s

batch  1:  14.54 GB -> 14.5 ms -> 68.8 tokens/s total, 68.8 per sequence
batch  8:  18.29 GB -> 18.3 ms -> 437  tokens/s total, 54.7 per sequence
batch 32:  31.18 GB -> 31.2 ms -> 1,026 tokens/s total, 32.1 per sequence

The same numbers can be placed on the roofline. Drag the batch size and watch the operating point: arithmetic intensity rises with batch because the weight read is shared, yet it never approaches the ridge point, because every sequence adds its own KV read (this toy counts only the weight-matrix operations).

This is the heart of the trade-off between throughput and p95 latency: batching makes each device far more cost effective, but each user in a large batch sees slower token pacing, and the batch is also what fills KV memory. The toy model leaves out compute limits (at batch 32 you are still well below the ridge in this example, so the ceiling is memory), attention kernels that are not perfectly bandwidth-efficient, fused kernels, speculative decoding and prefix reuse. Use it to reason about direction and magnitude, then measure.

Prefill, which processes the whole prompt in one pass, behaves oppositely: thousands of tokens multiply the same weights, intensity is high, and the matrix units are the limit. That is why the same device can be compute-bound while reading a 3,000-token prompt and bandwidth-bound while writing the answer. Peak theoretical FLOP/s describes only the first regime and describes it optimistically; delivered performance is always some fraction of it, set by tiling efficiency, memory stalls and non-matmul work.

Lower precision helps in three ways at once: weights occupy fewer bytes, fewer bytes cross the memory system per token, and matrix units can often perform more low-precision operations per cycle than high-precision ones. Each step down loses something, so the useful question is which format keeps enough accuracy for the task and is actually accelerated by your hardware.

A floating-point number splits its bits between an exponent, which sets dynamic range, and a mantissa, which sets resolution. FP32 has 8 exponent and 23 mantissa bits. FP16 has 5 exponent and 10 mantissa bits: good resolution but a small range, with a largest finite value of 65,504, so large activations or gradients overflow and training with it normally needs loss scaling. BF16keeps FP32's 8 exponent bits and has only 7 mantissa bits: it covers the same range as FP32 with coarser resolution, which is why it is the common training and serving format on hardware that supports it. FP8 formats exist in two common layouts, trading range against resolution (an E4M3 layout and an E5M2 layout); they are used with explicit per-tensor or per-block scale factors and require hardware and kernel support to be fast. Integer formats (INT8, INT4) store a small integer plus a scale.

A tiny quantisation example shows where accuracy goes. Take the weights [0.020, −0.500, 1.270, 0.003] with symmetric INT8 and one scale for the group: scale = 1.27 / 127 = 0.01. The quantised integers are [2, −50, 127, 0], which dequantise to [0.02, −0.50, 1.27, 0.00]. The big values are exact; the small 0.003 was rounded to zero. In real tensors, a few large outliers force a large scale and crush the small weights, which is why schemes use per-channel or per-group scales, keep sensitive layers in higher precision, or calibrate on sample data. Quality must be re-evaluated on your task, for instance the policy question set, because aggregate benchmark behaviour does not transfer reliably.

That last point is the practical hinge. Each combination of operation, data type, tensor shape and device generation needs a kernel: a hand-tuned or compiler-generated routine for exactly that case. If it exists the accelerator runs at a high fraction of peak. If it does not, the framework either raises an error, or silently falls back to a generic path that may be several times slower or that dequantises through a higher-precision intermediate and loses the benefit. A newly released data type may be supported on the newest device generation only. Before committing to a quantisation format, verify on the target device that the full serving path uses the optimised kernel, measure tokens per second and memory, and compare outputs against the BF16 reference.

Each family answers the two bottlenecks above with different priorities. None is a universal winner; the right comparison is always for a specific workload, on a specific software stack, at a specific scale.

GPUs began as graphics processors and became the default AI accelerator because their many-thread design maps well onto matrix work, because they remain programmable for new operations, and because a deep software ecosystem (kernel libraries, framework backends, profilers) has grown around them. The cost of flexibility is die area and power spent on general capability. They appear in the cloud, in workstations and in on-premises servers, and they are the platform most research code targets first, which is a real productivity advantage that does not appear in any peak-throughput comparison.

TPUs are accelerators designed around large systolic matrix units and a compiler-driven execution model, deployed as pods of tightly connected chips and accessed primarily through a cloud service. A systolic array moves operands between neighbouring multiply-accumulate cells so that data is reused without going back to memory, which is efficient for regular tensor programs. The model is then compiled ahead of time for the chip, which rewards static shapes and well-supported operations, and requires a framework path that targets the compiler. Programs written around dynamic, irregular control flow take more work.

NPUs are small accelerators integrated into phones, laptops and embedded systems. Their design goals are performance per watt and latency within a tight thermal envelope, typically for quantised inference of small models: image enhancement, speech recognition, on-device assistants. The software stack is usually vendor-specific, operator coverage is narrower than on datacenter devices, and memory is shared with the rest of the system. For the support assistant an NPU matters only if some part of the product runs on the customer's own device, for instance a local speech-to-text front end; it does not serve the shared model.

Custom ASICs are chips that a cloud provider or product company designs for its own workload mix, often for inference. The motivation is cost and efficiency at large volume and control over supply, in exchange for a large up-front engineering investment and a narrower, provider-specific software path. For a team renting capacity, the relevant questions are practical: does the serving framework support it, what do you lose in tooling, and what happens to portability if prices or availability change.

Across these families, two design axes explain most differences. One is how much is fixed in hardware versus left to a compiler or programmer (rigid systolic arrays and static scheduling on one side; many independent threads with dynamic scheduling on the other). The other is where the device sits: the datacenter optimises for throughput and interconnect scale, the edge for energy and latency. Software ecosystem sits behind both and is the factor most often underestimated.

Once a model, its optimiser state or its KV cache does not fit on one device, or one device is too slow, you need several, and the devices must exchange data. How fast depends on where the devices sit: on the same package or board, inside the same server, or in different servers.

Each step outward costs bandwidth and adds latency. The host interface is the narrowest path on the device; links between accelerators inside a node are designed to be much faster than that; and the network between nodes is usually slower again, shared with other traffic, and subject to congestion. Two numbers matter for any transfer: the bandwidth (how fast a large message flows) and the latency(what a tiny message costs), and the balance between them determines which parallel strategy is feasible.

Tensor parallelism splits individual matrix multiplications across devices and requires an all-reduce (or equivalent) after roughly every layer or layer sub-block; it is communication-heavy and latency-sensitive, so it is normally confined to devices on the same fast fabric. Data parallelism keeps a full replica per device and synchronises gradients once per step; the messages are large but infrequent and can overlap with the backward pass. Pipeline parallelism sends activations between stages and tolerates slower links but creates idle time (bubbles) in the schedule. See Parallel Processing for the strategies themselves; here the hardware point is that topology limits which of them you can afford.

Two toy communication estimates

A ring all-reduce over N devices moves about 2(N−1)/N times the message size through each device’s link. Use illustrative links: 100 GB/s within a node and 12.5 GB/s (a 100 Gbit/s network port) between nodes.

(a) Data-parallel gradient sync, 7B model, BF16 gradients = 14 GB, N = 8:
    traffic per device = 2 x (8-1)/8 x 14 GB = 24.5 GB
    over 100 GB/s intra-node: 24.5 / 100  = 0.245 s
    over 12.5 GB/s network:   24.5 / 12.5 = 1.96 s  (if not overlapped with compute)

(b) Tensor-parallel decode, N = 4, hidden size 4096, batch of 32 tokens, BF16:
    message = 32 x 4096 x 2 B = 262,144 B
    ring traffic per device = 2 x 3/4 x 262,144 = 393,216 B
    2 all-reduces per layer x 32 layers = 64 per decode step
    bandwidth cost = 64 x 393,216 B = 25.2 MB -> 0.50 ms at 50 GB/s effective
    latency cost   = 64 x 10 us (assumed per-collective latency) = 0.64 ms

The two cases lead to different engineering conclusions. In (a), faster links or overlapping communication with computation hides most of the cost. In (b), each message is tiny, so making the pipe wider barely helps; what matters is the number of collective operations and their fixed latency, which is why a per-token collective over a network hop would be unattractive for an interactive service even though the byte count looks small. Sizing a deployment by multiplying one device's throughput by the device count ignores exactly this term: communication-heavy strategies scale sub-linearly, and scaling efficiency should be measured, not assumed.

An accelerator draws power continuously, a large fraction of it even when lightly loaded, and that power becomes heat that must be removed. Three constraints follow. A server’s power supply and a rack’s feed set how many devices can be installed; the cooling system sets how much of the rated power can be sustained without the chip lowering its clock; and the facility’s total power budget sets the size of a cluster. Illustratively, if a node with 8 accelerators draws 8 × 500 W plus 1 kW for host and networking, that is 5 kW; a rack limited to 40 kW holds 8 such nodes. Density is bounded by watts and by the cooling that can remove them, not by the physical space.

Thermal throttling is the visible side effect: if cooling cannot keep up, clocks are reduced and delivered throughput drops below what a short benchmark showed. A benchmark that runs for 30 seconds may not reflect an hour of sustained load in a warm aisle.

The purchase or rental price per hour is therefore the beginning of the cost calculation, not the end. What you pay for is useful tokens, and idle time is billed as well. With an illustrative price of $4 per device-hour and a sustained 400 tokens/s while busy (1.44 million tokens per busy hour):

tokens per hour at 100% busy = 400 x 3600 = 1.44 M
$4.00 / 1.44 M tokens  = $2.78 per million tokens   (100% utilised)
$4.00 / (1.44 M x 0.70) = $3.97 per million tokens   ( 70% utilised)
$4.00 / (1.44 M x 0.35) = $7.94 per million tokens   ( 35% utilised)

A cheaper-per-hour device with half the throughput is not cheaper per token, and a faster, more expensive device that sits half idle is not either. Efficiency also includes energy per token and the operational cost of failures and software effort. The comparison between renting hosted API capacity (you pay per token and the provider carries idle cost) and running your own fleet (you pay per hour and carry idle cost yourself) is developed in Cost Optimization.

Nobody programs matrix units directly. A request to run a model passes through a layered stack, and every layer has its own versions and its own compatibility rules.

The framework expresses the model as operations on tensors. A compiler or runtime may rewrite that graph, fusing adjacent operations so that intermediate results stay in SRAM instead of round-tripping through HBM, and plans memory. Each remaining operation is then dispatched to a kernel from a vendor or open library, or one generated on the fly. The driver exposes memory allocation, copies and kernel launch to the operating system. Failures at any layer look similar from the top (an exception, or a slow job), which is why diagnosis has to be systematic.

The deployment lesson is to treat the stack as a version-pinned artifact. A container image holds the framework, libraries and user-space runtime; the host holds the kernel-mode driver; the two must be compatible, and the compatible range is defined by each vendor’s documentation. Upgrade them deliberately, test a representative model after each change, and keep the last known-good image available for rollback. Portability across accelerator families is rarely free: a model that runs on one platform may need different kernels, different quantisation formats and different parallelism settings on another, so lock-in should be assessed as the engineering cost of that move rather than as a binary property.

A single “best accelerator” does not exist because the three regimes stress different characteristics. For AI inference hardware, the priorities are memory bandwidth, KV capacity and batching within a latency target, rather than the sustained arithmetic that training favours. The three are worth separating with concrete scenarios.

Training runs for days or weeks on many devices. What dominates is sustained arithmetic on large batches, memory capacity for the 16-bytes-per-parameter state described earlier, high-bandwidth links for gradient or tensor synchronisation, and fault tolerance, because over thousands of device-hours something will fail and the job must resume from a checkpoint. Latency of an individual step does not matter; end-to-end throughput and reliability do.

Online inference serves many short requests under a latency target. For the support assistant, the user sees time to first token and then token pacing. Memory bandwidth, KV capacity, and the ability to batch without violating p95 latency matter most. A device that is excellent for training may be wasteful for low-concurrency serving of a small model, and a modest device with good bandwidth and enough memory may serve it economically.

Edge and on-device inference runs small, usually quantised models under strict power, thermal and cost limits, often with intermittent connectivity and privacy requirements that favour keeping data local. Energy per inference and the maturity of the vendor’s toolchain for the specific operators are the deciding factors. A 4-bit 3-billion-parameter model that needs 1.5 GB of raw weights (3e9 × 0.5 bytes) may fit a laptop NPU’s shared memory while a 70-billion-parameter model never will.

Note also that not every application operates accelerators at all. An application that calls a hosted API leaves the whole accelerator layer, including its failures, with the provider, and is billed per token. Self-hosting is justified by data-control, latency-locality, customisation or cost at scale, not by default.

The team decides to self-host the 7B model for the policy-question path. Their load test shows (illustrative): peak 40 concurrent sequences, an average of 3,000 tokens per sequence (about 2,500 for system prompt plus retrieved passages and the question, and 500 generated), and a p95 time-to-first-token target of 1.5 seconds. Using the KV formula and the illustrative architecture from earlier, work through the decision.

1. 1

   Model compatibility

   Confirm the serving engine supports this model architecture, attention variant and chosen quantisation on the candidate devices. If it does not, nothing else matters.
2. 2

   Memory headroom

   KV per sequence at 3,000 tokens: 131,072 × 3,000 = 0.393 GB. For 40 sequences: 15.73 GB. Add 14 GB of BF16 weights and an assumed 3 GB of runtime and workspace: 32.7 GB. A 24 GB device fails outright (and also fails with 1-byte weights: 7 + 15.7 + 3 = 25.7 GB). A 48 GB device leaves 15.3 GB headroom; an 80 GB device leaves 47.3 GB, which could take longer contexts or more concurrency.
3. 3

   Throughput and p95

   Use the toy decode model only to rank options, then benchmark. With the hypothetical 1 TB/s device a batch of 40 reads about 14 + 40 × 0.393 = 29.7 GB per step: roughly 30 ms per step, or about 34 tokens/s per user as a ceiling, comfortably above reading speed. A device with half the bandwidth would halve that. The real p95 must be measured under the real mix, including prefill of 2,500-token prompts.
4. 4

   Availability and failure recovery

   One device is a single point of failure and a cold start (loading 14 GB) takes time. A second replica gives redundancy and rolling updates; this doubles cost, so decide the SLO first. Autoscaling and routing are covered in Model Serving.
5. 5

   Developer effort and lock-in

   A well-supported platform with a mature engine may cost more per hour but save weeks. A less common platform may be cheaper per token but may lack a kernel for your attention variant or quantisation format. Price the engineering time, and record what a migration would involve.
6. 6

   Energy, cost and the API alternative

   Compute cost per useful token at expected utilisation (see the cost-per-million-tokens calculation above). If average utilisation would be 10% because traffic is bursty, a hosted API billed per token may be cheaper and operationally simpler. Revisit when sustained traffic justifies dedicated capacity.

The outcome of this example, under its assumptions, is: a 48 GB-class device (or larger) with two replicas, BF16 or validated 1-byte weights to create extra KV room, and a hosted-API fallback for overflow. The point is not the particular verdict. It is that the decision came from the KV formula and a latency estimate that anyone can check, rather than from a peak-FLOPS chart.

The opening puzzle can now be answered. Consider the same 7B BF16 model on a 24 GB device with an engine configured for up to 32 concurrent sequences of 4,096 tokens. The weights are 14 GB, so it seems to fit with 10 GB to spare. Under load at the 14th long request, the service dies with an out-of-memory error.

1. 1

   Symptom and where it appears

   OOM only under concurrency and long prompts, not with a single request. That pattern already suggests memory that grows with traffic, rather than a model that does not load.
2. 2

   Separate allocated from used

   Ask for three numbers: memory the framework has tensors in, memory the framework has reserved from the driver, and memory the device reports as in use. Reserved is usually larger than tensor memory because allocators cache blocks; the device total includes the runtime’s own context. Tools in the common ecosystems expose all three (an nvidia-smi query for device totals, framework allocator statistics for the others).
3. 3

   Account for every term

   Weights 14 + context 1.5 + workspace 1.0 leaves 7.5 GB for KV. At 0.537 GB per 4,096-token sequence that is 13 sequences (13.97 rounded down), so the 14th triggers failure; fragmentation makes it fail even earlier in practice. The configured limit of 32 sequences could never have been served.
4. 4

   Pick the right lever

   Options: lower the maximum concurrent tokens so admission control queues work instead of crashing; shrink the context budget for retrieved passages; use 1-byte weights (frees 7 GB, so room for 27 sequences by the same arithmetic); quantise the KV cache if quality allows; move to a larger device; or shard the model across two devices. Each has a quality, latency or cost price.
5. 5

   Verify and guard

   Re-run the load test at the new limit with a safety margin (here 24 of a theoretical 27 sequences), and add an alert on memory headroom and on rejected admissions so the next change in prompt length shows up in a dashboard instead of an outage.

GB = 1e9  # decimal gigabytes

def kv_bytes_per_token(layers, kv_heads, head_dim, bytes_per_elem):
    return 2 * layers * kv_heads * head_dim * bytes_per_elem  # K and V

def max_sequences(device_gb, weights_gb, overhead_gb, kv_per_token, tokens_per_seq):
    free = (device_gb - weights_gb - overhead_gb) * GB
    return int(free // (kv_per_token * tokens_per_seq))

kv = kv_bytes_per_token(layers=32, kv_heads=8, head_dim=128, bytes_per_elem=2)
print(kv)                                        # 131072
print(max_sequences(24, 14, 2.5, kv, 4096))      # 13  (BF16 weights)
print(max_sequences(24, 7, 2.5, kv, 4096))       # 27  (1-byte weights)

Memory is the most frequent failure, but the stack has others, and each has a different diagnostic path.

Incompatible or missing kernel. The model loads and answers, but tokens per second is a fraction of what the data sheet suggests, or the engine refuses to start with an unsupported-operation error. Check the engine’s startup logs for fallback warnings, profile one decode step and look at which kernels dominate, and compare the data type and shapes against the library’s documented support for your device generation. The remedy is usually a different quantisation format, a different attention backend, or an upgrade of the pinned stack, validated against a reference output.

Interconnect saturation. Throughput falls as you add devices, or per-token latency grows with the tensor-parallel degree. Compare achieved collective bandwidth with the link’s nominal rate, check whether the devices you were given share a fast fabric (placement matters: four devices spread across two nodes behave very differently from four in one), and look for the latency-dominated pattern from the toy calculation above. The remedy is placement, a lower parallelism degree, overlap, or a different strategy.

Device failure. A device reports uncorrectable memory errors, falls off the bus, or runs throttled. Serving should detect this through health checks and device telemetry, drain the affected replica, and let the router send traffic elsewhere; a lost device in a tensor-parallel group takes the whole group down, which is another reason to keep the parallel group small. For training the same event means resuming from the last checkpoint, so checkpoint frequency is an explicit trade between overhead and lost work.

With the accelerator now understood as a bundle of memory, arithmetic, links and software, the next question is what happens when you string thousands of them together for a single job that must run for weeks. That is a distributed-systems problem with its own failure modes.

1. 1

   A model can be slow for two structural reasons: data movement (decode is limited by reading weights from memory, not by arithmetic) and capacity (weights, KV cache, activations and runtime overhead must all be resident).

2. 2

   Weight memory is parameters × bytes per parameter: a 7-billion-parameter model needs about 14 GB in 16-bit weights, and training with the Adam recipe needs roughly eight times that.

3. 3

   The KV cache is a variable cost per concurrent token, so it belongs to traffic rather than to the model; in the page's example it is larger than the weights, which is why capacity planning has to budget it separately.

4. 4

   Single-stream decode latency is bounded by bytes of weights divided by memory bandwidth. Batching shares each weight read across sequences, which raises throughput but costs p95 latency; prefill behaves the other way and is compute-bound.

5. 5

   Lower precision saves bytes and bandwidth only if the kernels for that data type, shape and device exist; a missing kernel shows up as tokens per second far below the estimate.

6. 6

   Compare devices per token, not per hour, pin the driver, runtime and framework versions as one artifact, and diagnose bottom-up: device health and thermals, driver, kernels, interconnect, then the application.