CPUs vs GPUs vs TPUs

Watch a lone CPU worker struggle against a workload built for thousands, then see how GPU and TPU architecture make massive parallel computation possible.

GPU vs TPU vs CPU: which is used for AI, and why? All three can run AI, and the right one depends on the workload rather than a universal winner. CPUs run small models, preprocessing and control logic, while GPUs and TPUs accelerate the throughput of the dense matrix math that dominates training and large-model inference. Neural networks spend most of their time multiplying large matrices, and a matrix product is made of millions of small, independent multiply-and-add jobs. A CPU is built to finish one thread of unpredictable work quickly: a few strong cores, large caches, branch prediction. A GPU is built to finish a very large number of similar jobs per second: thousands of simple lanes executing the same instruction on different data, fed by very high-bandwidth memory. A TPU goes one step further for one pattern: Google's matrix-oriented accelerator wires multiply-accumulate cells into a grid so that operands are passed from cell to cell instead of being fetched again and again, and it relies on a compiler to map the whole model onto that grid.

That is the short answer, but it hides the part that decides real systems: arithmetic is rarely the limit. Whether a model runs fast depends on how many bytes must be moved for each operation, how much memory the device has, which numeric precision the hardware supports, and how often data crosses the link between host and accelerator. This chapter builds those ideas from one matrix multiplication, then uses them to make hardware decisions for four very different workloads. Every device figure below is an illustrative assumption chosen so the arithmetic is easy to follow; they are not specifications or benchmark results for any product.

Take a transformer layer processing 512 tokens with hidden size 4096 (illustrative sizes). The tokens form a matrix X of shape 512 x 4096. The layer's query projection multiplies it by a weight matrix W of shape 4096 x 4096 to produce another 512 x 4096 matrix. Attention scores, the output projection and the feed-forward block are further products of the same kind. A model with dozens of layers repeats this for every token it processes, so the performance of an LLM is, to a first approximation, the performance of matrix multiplication plus the cost of moving the matrices to where the arithmetic happens. For the model-level picture see Transformer Basics.

To see what the hardware is asked to do, shrink the problem. Multiplying A (m x k) by B (k x n) produces C (m x n), where every output element is a dot product:

C[i][j] = sum over t = 0..k-1 of A[i][t] * B[t][j]

each output element:  k multiplies + (k - 1) adds  ~=  2k floating-point operations
number of outputs:    m * n
total work:           ~= 2 * m * n * k   FLOPs

The property that matters is independence. C[0][0] and C[7][3] read overlapping inputs but neither waits for the other. A device with enough arithmetic units can compute huge numbers of outputs at once, and the order in which they finish does not change the answer. A workload full of such independent, regular, data-parallel work is what GPUs and TPUs are designed to eat. A workload full of decisions ("if this record is invalid, take a different path") is not, and that is the territory CPUs own.

A server CPU core is an engine for making one instruction stream fast even when the stream is unpredictable. It reorders instructions, guesses which way a branch will go before the condition is known, and keeps the data it expects to need in a hierarchy of caches (small and fast close to the core, larger and slower further out) so that most loads never reach main memory. A large fraction of the silicon is this machinery for hiding latency, not arithmetic. That is exactly what you want for parsing an HTTP request, walking a permissions tree, running a retrieval filter or deciding whether to call a tool.

CPUs are not scalar-only. Modern cores include SIMD (single instruction, multiple data) units: one instruction operates on a register that holds several numbers. A 256-bit vector register holds eight FP32 values, so one multiply-add instruction performs 8 multiplies and 8 adds. A purely illustrative server CPU with 8 cores, 2 vector multiply-add units per core, 8 FP32 lanes, 2 FLOPs per lane per cycle and a 3 GHz clock has a theoretical peak of 8 x 2 x 8 x 2 x 3 x 10⁹ = 768 GFLOPS. That is a ceiling, not a delivery promise: real matrix code on a CPU gets a fraction of it once caches, memory bandwidth and threading overheads enter, and the transformer-inference examples later in this chapter assume only 50 to 100 GFLOPS delivered. Libraries that tile matrices to fit caches close much of the gap, which is why a well-vectorised CPU is a legitimate inference target for small models.

Even in a GPU-heavy system the CPU stays on the critical path. It authenticates the request, applies tenant filters, tokenises text, runs the retrieval query, schedules requests into batches, launches accelerator work, streams tokens back and writes audit records. When a GPU sits idle at low utilisation, the cause is frequently CPU-side: slow tokenisation, a single-threaded scheduler or data loading that cannot keep up.

A GPU makes the opposite trade. It spends its silicon budget on arithmetic lanes and on memory bandwidth, and removes most of the speculation machinery. Lanes are grouped: on NVIDIA hardware a group of 32 threads executes in lock-step as a warp (AMD calls the equivalent a wavefront, with a width that depends on the architecture; check the vendor's programming guide for your device). Many warps are grouped into a block that runs on one streaming multiprocessor (SM), which has its own register file and fast on-chip shared memory. This style is called SIMT: single instruction, multiple threads. Each thread appears to have its own program counter, but the hardware issues one instruction to the whole warp at a time.

Three consequences follow, and each corrects a common wrong picture. First, a GPU hides memory latency by switching warps rather than by caching and speculating. When one warp waits for data, the scheduler runs another that is ready, so the GPU needs many more runnable threads than lanes to stay busy. A tiny problem leaves most of it idle. Second, lock-step has a price: if lanes in a warp take different branches, the hardware runs both paths with some lanes masked off, so branch-heavy code wastes lanes (warp divergence). Third, the unit of programming is a kernel: a function launched over a grid of threads. Each launch has a fixed overhead and takes a non-zero time to schedule, so a model that issues thousands of tiny kernels can be limited by launch overhead rather than arithmetic.

Modern GPUs also contain dedicated matrix units (NVIDIA calls them tensor cores) that multiply small tiles, for example 16 x 16 blocks, in a single operation, usually at reduced precision with higher-precision accumulation. Libraries tile a large multiplication into these blocks, stage tiles in shared memory so each loaded value is reused many times, and let the SMs work through the tiles in parallel.

A Tensor Processing Unit is a Google-designed application-specific integrated circuit for machine learning. Google's Cloud TPU documentation describes the design goal as hardware for the large matrix operations found in ML workloads, with on-chip high-bandwidth memory, and states that code on a TPU is compiled by the XLA (accelerated linear algebra) compiler, which compiles the linear-algebra, loss and gradient parts of the framework graph into TPU machine code while the rest of the program runs on the host (Cloud TPU introduction). The matrix unit (MXU) it describes is a 128 x 128 systolic array. TPU generations differ in sizes, memory, precision and interconnect, so treat any specific number in a blog post as belonging to one generation and check the current documentation for the one you would rent.

The idea of a systolic array is that data pulses through a fixed grid like blood through the body. Each cell holds one weight. An activation enters a row from the left edge, moves one cell right on every clock tick, and at each cell is multiplied by that cell's weight and added to a partial sum that is passed downward. Values are fetched from memory only at the edges; once inside the array each value is used by an entire row or column of cells before it leaves. That reuse is the point: it avoids the cost that dominates many accelerators, which is reading the same operand from memory repeatedly.

The trade is flexibility. A grid of multiply-accumulate cells is superb at dense matrix products and poor at arbitrary control flow. The same Google documentation lists TPUs as a good fit for models dominated by matrix computation and long training runs with large effective batch sizes, and as a poor fit for programs with frequent branching, many element-wise operations, workloads needing high-precision arithmetic, or custom operations inside the main training loop. Because XLA compiles the graph just in time for the first batch, the documentation also notes that tensors with dynamic shapes are poorly suited, and that when dimensions do not fill the matrix unit the compiler pads with zeros, which wastes compute and on-chip memory and can trigger out-of-memory errors. Practically: pick tensor dimensions that suit the hardware, keep shapes static, and accept that you are programming through a compiler.

Deployment is the other constraint. TPUs are available through Google Cloud (Compute Engine, GKE and managed platforms) rather than as a card you buy and slot into your own server, and chips are connected in groups called slices to scale a job. For a team on another cloud, or with an on-premises requirement, that settles the question before performance is considered. It also means a TPU decision is partly a framework decision: the mature paths go through JAX, TensorFlow or PyTorch bridges that target XLA.

Return to the formula with m = n = k = 1024. The count is the number of operations, not a measurement of any device, and it is the foundation for every comparison that follows.

outputs               = 1024 * 1024            = 1,048,576
per output            = 1024 multiplies + 1024 adds   (counting 1023 adds changes the total by 0.05 percent)
                        = 2 * 1024             = 2,048 FLOPs
total                 = 1,048,576 * 2,048      = 2,147,483,648 FLOPs
                      = 2 * 1024^3             = 2 * 2^30
                      ~= 2.147e9 FLOPs  (2.15 billion, decimal)

multiply-accumulate (MAC) count = 1024^3 = 1,073,741,824
one FMA instruction = 1 MAC = 2 FLOPs under this convention

Now give the number a time by dividing work by delivered throughput. If the CPU in the previous section delivers 100 GFLOPS (10¹¹ FLOP/s) on this product, it needs 2.147 x 10⁹ / 10¹¹ = 21.5 ms. If an accelerator delivers 100 TFLOPS (10¹⁴ FLOP/s) on it, the same work takes 21.5 microseconds, a thousand times less. These are two assumed delivered rates, not product claims. A vendor's headline TFLOPS figure is a theoretical peak for a specific precision under favourable conditions, typically with every matrix unit busy and all operands already on chip; the delivered number depends on the shape of the matrices, the precision, the library, memory traffic and power state.

Also count the bytes. In FP32 each matrix has 1024 x 1024 x 4 = 4,194,304 bytes (4 MiB, 4.19 MB decimal), so reading A and B and writing C moves 12,582,912 bytes (12 MiB, 12.58 MB decimal) at minimum. That gives the ratio that the next section turns into a picture: 2,147,483,648 FLOPs / 12,582,912 bytes = 170.7 FLOPs per byte for this product in FP32 (341.3 in BF16, where each element is 2 bytes). A lot of arithmetic per byte is the good case. It is also not the case for a lot of LLM serving.

Every operation needs two things: arithmetic units to do the operations and memory to deliver the operands. Arithmetic intensity is the ratio of FLOPs performed to bytes moved from memory, in FLOPs per byte. A device can sustain at most the smaller of two numbers: its peak compute rate, and its memory bandwidth multiplied by the intensity of the operation. Drawn on log axes this is the roofline: a rising slope where memory is the limit, a flat ceiling where compute is.

For a made-up accelerator with 200 TFLOPS peak and 2 TB/s memory bandwidth, the slope has value 2 TFLOPS per FLOP/byte (2 x 10¹² bytes/s x 1 FLOP/byte = 2 x 10¹² FLOP/s). The two lines meet at the ridge: 200 / 2 = 100 FLOPs per byte. An operation with intensity below 100 is memory-bound on this device: faster arithmetic would not help, only more bandwidth or fewer bytes would. Above 100 it is compute-bound.

Why a token is memory-bound

Consider generating one token with a dense weight matrix of size 4096 x 4096 in BF16 (2 bytes per value). With a batch of one, the operation is a matrix-vector product: 2 x 4096 x 4096 = 33.55 million FLOPs, while the weight matrix alone is 4096 x 4096 x 2 = 33.55 million bytes (the small vectors add about 0.05 percent). The intensity is almost exactly 1 FLOP per byte. On the illustrative device that is 1 x 2 TB/s = 2 TFLOPS, or 1 percent of the 200 TFLOPS peak. Each weight is loaded, used for exactly one multiply-add, and discarded.

Scale this to a whole model. Suppose an illustrative 7-billion-parameter model is stored in BF16: 7 x 10⁹ x 2 bytes = 14 GB (decimal; 13.04 GiB). Generating a token performs about 2 FLOPs per parameter, 14 GFLOPs, and must stream all 14 GB of weights from memory once. Time bounded by memory: 14 GB / 2 TB/s = 7 ms, so a single stream cannot exceed about 143 tokens per second no matter how fast the arithmetic is. Time bounded by compute: 14 GFLOPs / 200 TFLOPS = 0.07 ms. The memory term is 100 times larger. Doubling the device's TFLOPS changes nothing; doubling its memory bandwidth nearly doubles the token rate. This is why peak TFLOPS alone cannot predict LLM token speed, and why memory bandwidth is a headline figure for inference hardware. (The key-value cache adds more reads per token; that is examined in Inference Infrastructure.)

Data reuse is the escape

If 64 requests are decoded in the same step, the same weights are read once and used for 64 tokens, because the operation becomes a matrix product of the 4096 x 4096 weights with a 4096 x 64 block of activations. FLOPs rise 64x to 2.147 x 10⁹ while bytes rise only to 2 x 4096 x 4096 + 2 x 2 x 4096 x 64 = 34.6 million, an intensity of 62. The roofline now gives 62 x 2 = 124 TFLOPS: 62 percent of peak instead of 1 percent. Taking the whole model, one 7 ms step now yields 64 tokens, about 9,100 tokens per second in aggregate, while each user still sees roughly 143 tokens per second. That throughput-versus-latency trade is the basis of batching, and the same reuse principle explains matrix units, shared-memory tiling on GPUs and the systolic array's design: all of them are mechanisms for making each byte fetched from memory do more arithmetic.

The figure below lets you move the batch size yourself: it recomputes intensity, step time and token rates from the same formulas.

Large training batches and large prompt processing (prefill) sit on the compute-bound side; single-stream generation and many small operations such as layer normalisation and activation functions sit on the memory-bound side. A real kernel also loses to launch overhead, imperfect tiling and power limits, so the roofline is an upper bound on what a given intensity can reach, not a forecast.

Precision is a lever on all of the above. A number stored in fewer bits takes less memory and bandwidth, so memory-bound operations speed up almost in proportion, and many accelerators have matrix units that process low-precision inputs at several times the rate of FP32. Storage for the illustrative 7-billion-parameter model scales directly: 28 GB in FP32, 14 GB in 16-bit formats, 7 GB in 8-bit and 3.5 GB in 4-bit (decimal GB, weights only). A model that does not fit on one device at 16-bit may fit at 8-bit, which can change the whole hardware plan.

The formats trade differently. FP32 has 8 exponent bits and 23 mantissa bits. BF16 keeps the 8 exponent bits, so it has the same dynamic range as FP32 (largest finite value about 3.4 x 10³⁸), but only 7 mantissa bits, so adjacent representable numbers are far apart. FP16 keeps 10 mantissa bits, giving finer steps, but only 5 exponent bits: the largest finite value is 65,504, and tiny gradients can underflow to zero. That difference is why FP16 training usually needs loss scaling, while BF16 training often does not. Accumulation matters too: matrix units usually multiply low-precision inputs but add into a higher-precision accumulator, because summing thousands of terms in a 10-bit mantissa loses small contributions. The behaviour is easy to see on a laptop:

import numpy as np

print(np.float16(65504))                    # 65504.0  largest finite FP16
print(np.float16(70000))                    # inf      overflow (warns)
print(np.float16(2048) + np.float16(1))     # 2048.0   the +1 is absorbed: spacing at 2048 is 2
print(np.float32(2048) + np.float32(1))     # 2049.0   FP32 keeps it

Quantisation to integers

Integer formats such as INT8 map real values to a small grid using a scale: q = round(x / s), reconstructed as x' = s x q. If a tensor's values lie in [-12.7, 12.7], the scale is s = 12.7 / 127 = 0.1. The value 0.37 becomes q = round(3.7) = 4 and is reconstructed as 0.4, an error of 0.03; a value of 0.04 rounds to 0 and disappears. The scale is chosen to cover the largest values, so a few outliers force coarse steps for everything else, which is why outliers in LLM activations are a well-known difficulty and why techniques use per-channel or per-group scales. Quantisation is also not only a hardware switch: it changes model behaviour and must be evaluated on your task, an evaluation discipline discussed in Fine-Tuning for adapter releases and applicable here as well.

The accelerator's fast memory is separate from the host's. A tensor produced by the CPU lives in system RAM; before a GPU kernel can use it, it must be copied across a link (PCIe on most servers, a coherent chip-to-chip link on some systems, or no copy at all on devices with shared memory) into the device's memory. Results come back the same way. The link is much slower than the device's own memory: the GPU may read its HBM at terabytes per second, while the host link might sustain tens of gigabytes per second. For the illustration I assume 25 GB/s effective in each direction plus 10 microseconds of fixed overhead per copy call, and 8 microseconds to launch a kernel. These are invented round numbers, not a PCIe specification.

Take the 1024 x 1024 FP32 product again, with A and B on the host and the result needed on the host:

CPU_FLOPS   = 100e9    # assumed delivered CPU rate
GPU_FLOPS   = 100e12   # assumed delivered accelerator rate
LINK_BYTES  = 25e9     # assumed effective host<->device bytes/s
COPY_FIXED  = 10e-6    # assumed per-copy overhead (s)
LAUNCH      = 8e-6     # assumed kernel launch overhead (s)

def times(n):
    flops = 2 * n**3
    nbytes = 3 * n * n * 4            # A in, B in, C out, FP32
    cpu = flops / CPU_FLOPS
    xfer = nbytes / LINK_BYTES
    gpu = 3 * COPY_FIXED + LAUNCH + xfer + flops / GPU_FLOPS
    return cpu, gpu, xfer

print("   n      cpu(us)   gpu(us)  transfer share of gpu")
for n in (128, 256, 512, 1024, 4096):
    cpu, gpu, xfer = times(n)
    print(f"{n:5d} {cpu*1e6:12.1f} {gpu*1e6:9.1f} {100*xfer/gpu:8.0f}%")

Two lessons sit in the output. At n = 128 the product is 4.2 million FLOPs and 196,608 bytes: the CPU finishes in 42 microseconds, the accelerator needs 46 because 38 of those are fixed copy and launch overhead. For work this small the CPU, whose data is already next to it in cache, wins. At n = 1024 the accelerator is 38 times faster than the CPU in elapsed time, yet 89 percent of its time is spent on the link and its effective throughput is 2.147 x 10⁹ / 562.8 x 10^-6 = 3.8 TFLOPS, not the 100 TFLOPS the compute step alone suggests. Delivered application performance includes the movement.

Slide the matrix size in the figure below to see where the crossover happens under the same assumed rates.

The remedies follow from the model. Keep tensors resident on the device across many operations so the copy is amortised: a model's weights are uploaded once at load time, and only token IDs or activations that actually change cross the link. Copy in large batches rather than many small calls. Overlap transfers with compute using asynchronous copies from pinned (page-locked) host memory, so the link is busy while kernels run. And when a model is split across devices, the traffic between them (which uses faster interconnects than PCIe on many servers, but is still far slower than on-chip memory) becomes the next bottleneck; that is the subject of Parallel Processing.

Workloads sit at different places on the roofline. Training processes large batches, so each weight read is reused across many examples, making the matrix products compute-bound and rewarding peak throughput, low-precision matrix units and fast links between devices. It also needs memory for activations, gradients and optimizer state, not only weights. Interactive inference often serves small batches under a latency budget, which pushes it toward the memory-bound side and makes bandwidth, batching and scheduling the levers. Edge inference inverts the constraints again: the model is small, the power and thermal budget is a few watts, and the right hardware is a neural-processing unit or even the CPU already in the device. Pre- and post-processing (tokenising, parsing, filtering, ranking business rules) is branchy, sequential or I/O-bound, and belongs on CPUs.

So GPUs are common in AI because the most expensive operation is large dense matrix algebra, not because every AI feature needs one. The enterprise support assistant in this series is a good illustration of a mixed system.

The orchestrator, tenant filter and order-lookup tool never touch an accelerator. The embedding model might. The answer model runs on an accelerator that either belongs to your provider (you pay per token and operate nothing) or to you (you pay per GPU-hour and own utilisation, memory fit and failure handling). The first choice is a procurement decision about hardware; the rest of this chapter is what you need to make the second one. Remember also that the LLM does not know an order's status; the authorised tool does, and the model sees only what the orchestrator places into its context (Context Window).

Each decision below follows the same reasoning: estimate FLOPs and bytes, compare against a latency or completion-time target, then check memory fit and operational constraints. Numbers are invented but internally consistent, so you can redo them with measured values.

1. A small embedding service for retrieval

Suppose the retrieval stack uses an embedding model with 100 million parameters (illustrative). A forward pass costs about 2 x parameters x tokens FLOPs. A user query of 16 tokens costs 2 x 10⁸ x 16 = 3.2 x 10⁹FLOPs. On a CPU server delivering 50 GFLOPS on this workload (assumption; short sequences use the vector units poorly) that is 64 ms, so one server serves about 15.6 queries per second; two servers cover a 10 queries/s peak with headroom and no accelerator to operate. On a GPU delivering 20 TFLOPS the compute is 0.16 ms, but the request still pays link and scheduling overhead and the GPU would sit nearly idle at that arrival rate. Ingestion is a different story: embedding 1 million 512-token chunks costs 2 x 10⁸ x 512 = 1.024 x 10¹¹ FLOPs each, so 2.05 s each on the CPU (23.7 days for the set on one server) and 5.12 ms on the GPU (1.42 hours). The sensible split is CPU for live queries, a GPU borrowed for bulk re-indexing, and the choice is driven by duty cycle rather than by the model. See Embeddings for how the vectors are used.

2. Fine-tuning a large language model

Full fine-tuning of a 7-billion-parameter model with Adam in mixed precision stores BF16 weights (2 bytes), BF16 gradients (2), an FP32 master copy (4) and two FP32 optimizer moments (8): 16 bytes per parameter, or 7 x 10⁹ x 16 = 112 GB before counting activations. A single 80 GB device (illustrative capacity) does not hold that, so the plan needs either sharding the state across several devices or a parameter-efficient method such as LoRA, which freezes the 14 GB of base weights and trains small adapter matrices. Arithmetic is large-batch and compute-bound, links between devices matter, and a job that runs for hours justifies the cost and the engineering of a GPU or TPU cluster. TPUs are a candidate if the training stack is JAX or XLA-compatible PyTorch and the team is on Google Cloud; otherwise GPUs are the lower-friction path. The distributed mechanics are in Training Infrastructure.

3. A latency-sensitive classifier that is CPU-friendly

A ticket-routing model with 5 million parameters on 64-token inputs costs 2 x 5 x 10⁶ x 64 = 6.4 x 10⁸FLOPs, 12.8 ms at 50 GFLOPS delivered. With a 30 ms p95 budget that fits, and scaling out by adding CPU replicas is cheap, stateless and available everywhere. A GPU would add a copy, a launch and a queue, and, at batch 1, would leave the device mostly idle, so a GPU raises cost without lowering p95. The CPU choice holds while the model stays small, the traffic is steady and requests arrive one at a time; if the model grows 50 times or latency targets halve, rerun the numbers. Notice that this is an engineering argument, not a rule that CPUs are for small models.

4. An edge or mobile model

A 3-million-parameter model is 12 MB in FP32 and 3 MB in INT8 (weights, decimal). The constraints are memory footprint, battery, thermals and latency without network access. The device's NPU or DSP typically runs INT8 efficiently, but only for the operators its runtime supports; unsupported operators fall back to the CPU, and moving tensors between the two can erase the gain, the same transfer logic as above at a smaller scale. Quantisation is therefore not optional here, and the test is performance on the actual device after a sustained run, because phones throttle. No server-class accelerator enters the picture.

Having argued each case, a compact summary of what drove the choice may be useful, because it shows the pattern: the dominant constraint, not the model's fame, selects the hardware.

A single number such as "X tokens per second" or "Y TFLOPS" says almost nothing without its context. Compare hardware using the measured combination of the model and its size, the precision, the batch size and sequence lengths (input and output tokens separately), the memory used versus allocated, the achieved throughput, the latency distribution (p50 and p95, not only mean), device utilisation, power draw and, for decisions, the total cost per unit of useful work. A benchmark record that can be reproduced looks like this:

{
  "model": "7b-instruct-example",
  "precision": "bf16",
  "device": "accelerator-A (placeholder)",
  "input_tokens": 1500,
  "output_tokens": 300,
  "batch_size": 16,
  "throughput_output_tokens_per_s": 1500,
  "latency_ms": { "ttft_p50": 420, "ttft_p95": 900, "per_token_p95": 38 },
  "memory_gb": { "allocated": 78, "peak_used": 61 },
  "power_w_avg": 450,
  "hourly_cost_usd": 2.00
}

From such a record the economics follow. If the device is rented at $2.00 per hour (illustrative) and sustains 1,500 output tokens per second, the cost is 2.00 / (1500 x 3600) x 10⁶ = $0.37 per million output tokens. If real traffic keeps the device only 40 percent busy, the effective rate is 600 tokens per second and the cost becomes $0.93 per million. The hardware did not change; utilisation did, and that is the figure that a headline TFLOPS comparison never contains. Compare it with an API provider's published token price as a make-or-buy check, and see Cost Optimization for the broader trade-offs.

Method matters as much as the table. GPU kernels launch asynchronously, so a timer around a Python call can stop before the work finishes unless you synchronise the device first. The first iterations include compilation, memory allocation and cache warm-up, so discard them. Measure at your real batch size and prompt lengths, with an arrival process that resembles production rather than a closed loop that always keeps the queue full, and report latency percentiles together with throughput, because the same hardware can show high peak throughput and a poor p95.

import torch

assert torch.cuda.is_available(), "no CUDA device visible"
n = 1024
a = torch.randn(n, n, device="cuda", dtype=torch.float32)
b = torch.randn(n, n, device="cuda", dtype=torch.float32)

for _ in range(10):                      # warm-up: allocation, kernel selection
    a @ b
torch.cuda.synchronize()                 # wait for queued kernels

start, end = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
iters = 100
start.record()
for _ in range(iters):
    a @ b
end.record()
torch.cuda.synchronize()

ms = start.elapsed_time(end) / iters
print(f"{ms:.3f} ms/iter  ->  {2 * n**3 / (ms / 1e3) / 1e12:.2f} TFLOPS delivered")

Hardware choices show up in operations as a short list of recurring failures. Out-of-memoryis the most common. Device memory must hold the weights, the working activations and, for serving, the key-value cache; the 7-billion-parameter BF16 model above uses 14 GB for weights alone (decimal), so an 80 GB device (illustrative) has about 66 GB left for everything else, and a runtime that reserves memory up front will report that memory as allocated whether or not it is in use. Allocated is not used: monitor both and treat fragmentation, a burst of long prompts or an unexpectedly large batch as the usual causes of a crash that arrives hours after a healthy start.

Power and thermal limits do not crash anything; they slow it. A device that reaches its power or temperature limit lowers its clocks, so throughput decays over minutes and a short benchmark flatters it. Dense racks, failed fans and ambient temperature all matter, and on laptops and phones thermal throttling is the dominant effect. Watch clock speed and temperature next to utilisation.

Driver and runtime compatibility is the quiet one. A framework build targets a particular runtime library version, which requires a sufficiently recent driver on the host; containers must be given access to the device and the matching user-space libraries. After an OS or driver update the symptoms are "device not found", fallbacks to the CPU that run a hundred times slower without an error, or kernel errors for architectures the build does not include. Pin versions, record them in the benchmark and deploy through tested images. Availability is the commercial version of the same problem: quota, regional capacity and preemptible instances mean that the device you sized for may not be there when a replica restarts, so a fallback (smaller model, CPU path or hosted API) is part of the design.

1. 1

   Confirm the device is visible and usable

   In the failing process, check that the framework sees an accelerator, which driver and runtime versions it reports, and that a trivial allocation and kernel succeed. A silent CPU fallback is the first thing to rule out.
2. 2

   Separate capacity from behaviour

   Compare allocated and used memory over time. A steady climb points to a leak or fragmentation; a sudden step on one request points to a long input or oversized batch.
3. 3

   Check for throttling

   Plot clocks, temperature and power draw against throughput. Falling clocks with flat load means a thermal or power limit, not a software regression.
4. 4

   Locate the bottleneck type

   Low device utilisation with high CPU use suggests a host-side bottleneck (tokenising, scheduling, data loading). High utilisation with low TFLOPS suggests a memory-bound operation; compute against the roofline above.
5. 5

   Check data movement

   Count copies per request and their sizes. Many small copies or a repeated upload of weights or large inputs is the pattern that made the 1024 x 1024 example link-bound.
6. 6

   Decide the fallback

   Pre-define what happens when no device is available or memory is exhausted: queue, shed load, route to a smaller model or to a hosted API.

Everything above treats a device as the unit. Two numbers in the examples already point past it. The 112 GB fine-tuning footprint exceeded a single 80 GB device, and the weights of a larger model will not fit at all. And once a model is split across devices, the cost that dominated our small example, moving bytes, returns as communication between chips. Deciding how to partition a model and its data so that computation keeps ahead of communication is the subject of parallel processing, and the internal design of the accelerators themselves (memory stacks, interconnects, the precisions they support) is picked up in AI Accelerators.