Training vs Inference

Follow one user request backward through the hidden training phase and then forward through the serving path that produces a response.

Inference vs training comes down to this: training adjusts a model's weights to fit data, while inference applies the fixed, trained weights to a new input to produce an output. A spam classifier learns from labelled emails in training, then labels each incoming email at inference; the spam example below follows one email through both.

Training is the phase in which a model's learnable numbers, its parameters or weights, are adjusted so that its predictions fit data. Inference is the phase in which those numbers are held fixed and applied to a new input to produce a prediction: a label, a score, a number, or the next token of generated text. Training is a loop of many cheap-to-describe but expensive-to-run updates over a dataset; inference is a single computation, or for text generation a repeated one, run once per request. The distinction concerns model parameters. A surrounding application can still change state at inference time (a chat history grows, a cache fills, a database row is written) without any weight changing.

Everything below follows from that one split: why training needs memory for gradients that serving does not, why a service is judged on latency while a training job is judged on examples per second, and why a model can be good in the lab and bad in production. The scope is learned models. A rule-based system (see What Is AI?) has no training phase, and not every learned model is trained by gradient descent; the worked example here uses it because it is the dominant method for neural networks, and the placement of this lesson among the families is the subject of ML vs DL vs Generative AI.

An email provider ships a spam classifier. Last week a training job read several million labelled emails ("spam" or "not spam"), adjusted the model's weights, was evaluated on emails it had not trained on, and was saved as a file. Now a new message arrives: "FREE gift, claim now: http://…". The provider does notstart a training run for it. A service loads the saved file (once, at startup), converts the message into numbers using the same feature recipe used in training, multiplies and adds them through the model, and returns a probability. If the user marks the message "not spam", that click is stored as a new labelled example; it influences the model only if a future training run reads it.

To make this concrete, use a deliberately tiny model: logistic regression over two features. For each email, x₁ is the number of promotional trigger words and x₂ is 1 if the message contains a link, else 0. The learned parameters are two weights and a bias, w = (0.8, 1.2) and b = −2.5 (illustrative values, as if learned). The new email has three trigger words and one link, so x = (3, 1).

import math

w = (0.8, 1.2)         # learned parameters (fixed at inference)
b = -2.5               # learned bias
x = (3, 1)             # runtime input: features of THIS email

z = w[0]*x[0] + w[1]*x[1] + b     # 0.8*3 + 1.2*1 - 2.5 = 1.1
p_spam = 1 / (1 + math.exp(-z))   # sigmoid(1.1) = 0.7503
print(round(z, 4), round(p_spam, 4))   # 1.1 0.7503

Three different kinds of number appear in this tiny program, and keeping them apart prevents most confusion later. Parameters (w, b) are learned during training and stored in the model artifact. Inputs (x) arrive with each request and are never stored in the model. Hyperparameters (the learning rate, batch size, number of epochs, the decision threshold, regularization strength) are chosen by engineers and are not learned by the update rule they configure; they are tuned by comparing runs on validation data. A model with a billion parameters has the same three categories, just with more of the first.

Learning needs four ingredients. First, examples: pairs of an input x and, in supervised learning, a label or target y (spam = 1, not spam = 0). Second, a model: a function with adjustable parameters that maps x to a prediction ŷ. Computing ŷ from x and the current parameters is the forward pass. Third, a loss function L(ŷ, y): a single non-negative number measuring how wrong this prediction is, chosen so that smaller is better and so that it is differentiable with respect to the parameters. Fourth, an optimizer: the rule that turns "how the loss changes when each parameter changes" into an actual parameter change.

For the spam email above, suppose its true label is y = 0 (a newsletter the user wanted). The model said 0.7503, which is confidently wrong. The cross-entropy loss for a label of 0 is −ln(1 − p) = −ln(0.2497) ≈ 1.387. For this loss with a sigmoid output the derivative of the loss with respect to the score is p − y = 0.7503, so the gradient with respect to each weight is (p − y)·xᵢ: about 2.25 for w₁ and 0.75 for w₂. An optimizer that subtracts a small multiple of these would shrink both weights, making similar emails look less spammy. The next section runs exactly this mechanism with numbers small enough to follow by hand.

Use the smallest possible model: a single weight, no bias, and a regression target. The model is ŷ = w·x. The loss for one example is half the squared error, L = (ŷ − y)² / 2; the factor ½ exists only so the derivative comes out clean. The example is x = 2 with target y = 6, the starting weight is w = 1, and the learning rate (a hyperparameter) is η = 0.1. The ideal weight is obviously 3, but the algorithm does not know that; it only sees the loss and its slope.

1. 1

   Forward pass

   ŷ = w·x = 1 × 2 = 2.
2. 2

   Loss

   L = (ŷ − y)² / 2 = (2 − 6)² / 2 = 16 / 2 = 8.
3. 3

   Gradient (chain rule)

   L depends on w through ŷ, so dL/dw = (dL/dŷ)·(dŷ/dw) = (ŷ − y)·x = (2 − 6)·2 = −8. The negative sign says: increasing w would decrease the loss, and the slope is steep.
4. 4

   Update

   w ← w − η·(dL/dw) = 1 − 0.1 × (−8) = 1 + 0.8 = 1.8. Subtracting the gradient moves against the slope, i.e. downhill.
5. 5

   Check

   New forward pass: ŷ = 1.8 × 2 = 3.6; new loss = (3.6 − 6)² / 2 = 5.76 / 2 = 2.88. The loss fell from 8 to 2.88.

The loss fell because the step was small relative to the curvature: the loss as a function of w is L(w) = (2w − 6)²/2 = 2(w − 3)², a parabola with its minimum at 3, and a short move downhill lowers it. Run the same step again and again and the error shrinks geometrically. Write the error as e = ŷ − y. One update gives w' = w − η·e·x, so e' = e·(1 − η·x²); here the factor is 1 − 0.1 × 4 = 0.6. The error goes −4, −2.4, −1.44, … and the loss, which is proportional to e², shrinks by 0.36 each step.

w, x, y, lr = 1.0, 2.0, 6.0, 0.1

for step in range(1, 6):
    y_hat = w * x                    # forward pass
    loss  = (y_hat - y) ** 2 / 2     # loss
    grad  = (y_hat - y) * x          # dL/dw
    print(f"step {step}: w={w:.4f} y_hat={y_hat:.4f} loss={loss:.4f} grad={grad:.4f}")
    w -= lr * grad                   # optimizer update

# step 1: w=1.0000 y_hat=2.0000 loss=8.0000 grad=-8.0000
# step 2: w=1.8000 y_hat=3.6000 loss=2.8800 grad=-4.8000
# step 3: w=2.2800 y_hat=4.5600 loss=1.0368 grad=-2.8800
# step 4: w=2.5680 y_hat=5.1360 loss=0.3732 grad=-1.7280
# step 5: w=2.7408 y_hat=5.4816 loss=0.1344 grad=-1.0368

Two cautions keep this honest. First, this is one gradient step on one example in a problem with a single minimum. Real models have millions to billions of parameters and non-convex losses; a step lowers the loss locally, and there is no guarantee of reaching a global minimum, only a hope of reaching a useful one. Second, the learning rate matters. In this example the error factor is 1 − 4η. With η = 0.5 the factor is −1: w bounces 1 → 5 → 1 → 5 forever and never improves. With η = 0.6 the factor is −1.4: the weight goes 1 → 5.8 → −0.92 → 8.49, and the loss grows each step. A learning rate that is too large makes training diverge, and in deep networks the same effect appears as exploding gradients. The boundary (η < 0.5 here) depends on the curvature, which is why it is a tuned hyperparameter rather than a constant of nature.

Drag the learning rate below to watch the same six-step update from w = 1 converge, bounce or diverge.

Real training never feeds one example at a time or the whole dataset at once. It shuffles the dataset and processes it in mini-batches: the loss is averaged over, say, 256 examples and the gradient of that average is used for one update. Averaging reduces the noise of a single example while keeping each step cheap enough that thousands of steps are affordable. One full pass over the dataset is an epoch; training typically runs many (although very large pretraining runs may see each example only once or a few times). Shuffling each epoch prevents the model from learning the order of the data.

A mini-batch changes nothing conceptually. Take three examples of the same model, (x, y) = (2, 6), (1, 3.2), (3, 8.7), with w = 1. Per-example gradients (wx − y)·x are −8, −2.2 and −17.1; their mean is −27.3 / 3 = −9.1, and the update is 1 − 0.1 × (−9.1) = 1.91. The weight reflects all three examples at once, whereas the single-example step above moved to 1.8 using only the first.

best_val = float("inf")
for epoch in range(max_epochs):
    model.train()                          # training mode: dropout on, batch-norm uses batch stats
    for x, y in shuffled_batches(train_set, batch_size):
        y_hat = model(x)                   # forward pass
        loss = loss_fn(y_hat, y)           # scalar loss on this batch
        optimizer.zero_grad()              # clear gradients from the previous step
        loss.backward()                    # backward pass: fills p.grad for every parameter
        clip_grad_norm(model.parameters(), max_norm)   # optional guard against exploding gradients
        optimizer.step()                   # update parameters using the gradients

    model.eval()                           # evaluation mode: dropout off, running statistics
    with no_grad():                        # no graph is built; gradients are not computed
        val_loss = mean(loss_fn(model(x), y) for x, y in batches(val_set))

    if val_loss < best_val:
        best_val = val_loss
        save_checkpoint(model, optimizer, epoch)   # weights AND optimizer state, to resume or roll back
    elif epochs_since_best >= patience:
        break                              # early stopping

Note the separation the loop enforces. During training the model is in a mode in which stochastic layers (dropout) are active and statistics (batch normalization) are computed from the current batch, and the framework records the operations needed for the backward pass. During evaluation those behaviours are switched off and gradients are not tracked, because evaluation measures the model rather than changes it. Forgetting model.eval() before serving is a classic bug: predictions become slightly random and depend on what else is in the batch. Checkpoints save the optimizer state as well as the weights because resuming Adam without its moment estimates is not the same as continuing the run.

A model with enough capacity can memorize its training examples, giving near-zero training loss and poor performance on new data. The goal is generalization: low error on inputs the model has never seen. That is why data is split three ways. The training set drives parameter updates. The validation set is used during development to choose hyperparameters, compare architectures and decide when to stop; because decisions are made on it, it gradually becomes tuned to. The test set is touched rarely, at the end, to produce an honest estimate of performance. If the same data serves for training and for the final number, the number is optimistic by construction.

In the figure (illustrative numbers), training loss keeps improving but validation loss stops improving after epoch 5 and then climbs: the model is spending capacity on quirks of the training set. Early stopping keeps the checkpoint from the best validation epoch. Regularization attacks the cause: weight decay penalizes large weights, dropout randomly disables units during training, data augmentation widens the training distribution, and simply collecting more data narrows the gap.

The ways validation numbers lie are mostly about leakage, information from outside the training boundary reaching it. Near-duplicate emails in both training and validation sets inflate the score; so does splitting a time series randomly rather than by time (the model sees the future), computing normalization statistics over the full dataset before splitting, or a feature that is only available because the label was already known. A leaky evaluation looks like a great model right up to the moment it meets production. Finally, a good test score certifies the model only on the distribution the test set came from. Distribution shift, when real inputs differ from the training data (new spam campaigns, a new product category, a new language), degrades accuracy with no change to the model at all, which is a major reason monitoring is part of the lifecycle below.

Training from scratch begins with randomly initialized weights and learns from a large dataset: the expensive phase. For large language models it is called pretraining, and its objective is typically next-token prediction (see Next-Token Prediction): given the preceding tokens, assign probability to the next one, with the loss comparing that distribution to the actual token. Because the labels come from the text itself, no human annotation is needed per example, which is what makes the scale possible. The transformer forward computation that produces those probabilities is described in Transformer Basics, and the text is first converted to the integer units described in Tokens Explained.

Fine-tuning starts from pretrained weights instead and continues training on a smaller, targeted dataset: classified support tickets, instruction-following examples, a company's tone. It is still training: the loop above, with a backward pass and an optimizer, and the weights change. Because updating every weight of a large model needs the full training memory budget, parameter-efficient methods such as LoRA freeze the original weights and train small low-rank matrices added to some layers, shrinking the number of trainable parameters and thus the gradient and optimizer memory. The full treatment, including when fine-tuning is the wrong tool, belongs to the Fine-Tuning lesson.

Inference then runs the finished (or adapted) model. Prompting a model, however cleverly, is inference: giving it examples in the prompt changes its output through the input, not its weights, and when the conversation ends nothing about the weights has changed. What does change in a deployed application is the state around the model: conversation history sent with each request, documents retrieved for the prompt, a cache, a user profile in a database. That state can be written during a request, and it can look like "the assistant remembered," but it is application data re-presented as input. It becomes learning in the technical sense only if it is later turned into training data and a training run is performed.

The inference path has the same forward computation as the training loop but nothing after it that learns. In the spam service: a request carries the raw email; preprocessing turns it into the exact numeric form the model was trained on (feature extraction here; tokenization and ID lookup for a language model); the forward pass runs with weights already in memory; postprocessing applies the threshold or decodes output IDs back to text; the service responds. Loading the model file from storage into memory and onto the accelerator happens once at startup (a slow "cold start" if it is repeated), not per request.

How many forward passes a request needs depends on the task. A classifier or an embedding model usually needs one: input in, answer out. An autoregressive language model needs many: one forward pass over the whole prompt (the prefill, which also yields the first output token) and then one forward pass per additional generated token, each feeding the previous output back in, until a stop condition. Generating 64 tokens is therefore roughly 1 prefill plus 63 further decode steps. Serving engines keep the attention keys and values of earlier tokens in a KV cache so each decode step does not recompute the whole prefix; that cache is runtime state, not learned weights, and it will matter in the memory budget below. The detail of serving it is in Inference Infrastructure.

Take a model with 1 billion parameters. Stored in FP16 (2 bytes per parameter) its weights occupy 1 × 10⁹ × 2 bytes = 2 × 10⁹ bytes = 2 GB (decimal gigabytes; binary GiB would give about 1.86). That is the lower bound for inference, and it is only the lower bound.

Training must also hold, for every parameter, a gradient, and for most optimizers extra state. Under one common mixed-precision accounting with the Adam optimizer: FP16 weights (2 bytes) + FP16 gradients (2 bytes) + an FP32 master copy of the weights (4 bytes) + Adam's first moment (4 bytes) + second moment (4 bytes) = 16 bytes per parameter, so 16 GB before a single activation. Activations, the intermediate outputs of every layer saved for the backward pass, scale with batch size and sequence length; six gigabytes in the figure is an illustration, and recomputing activations during the backward pass ("activation checkpointing") trades compute for memory. Inference can discard each layer's output as soon as the next layer has consumed it.

Inference has its own growing term. For a transformer the KV cache stores, per token and per layer, one key and one value vector for each key-value head. Per token that is 2 × layers × kv_heads × head_dim × bytes. For an illustrative configuration of 24 layers, 16 key-value heads of dimension 64, in FP16: 2 × 24 × 16 × 64 × 2 = 98,304 bytes per token. A 4,096-token sequence needs 98,304 × 4,096 = 402,653,184 bytes ≈ 0.40 GB; eight concurrent sequences need about 3.22 GB, more than the weights' own 2 GB. The configuration is hypothetical; real models differ in layer count and in how many key-value heads they use. The principle holds: serving memory is weights plus a cache that scales with concurrency and context length.

The 16-byte figure is an accounting for one setup, not a law. An optimizer without momentum state (plain SGD) needs far less; 8-bit optimizer states, sharding the optimizer state across devices, or LoRA (where only small adapter matrices receive gradients) all change the total. Inference can shrink too: quantizing weights to 8 or 4 bits cuts the weight term, at some cost in accuracy that must be measured. The practical consequence is that a model comfortable on one serving GPU may need several accelerators, or a sharded optimizer, to train. Distribution across devices is the subject of Training Infrastructure.

Each phase has a different notion of "fast." Online inference serves a person or a calling service, so it is judged on the time from request to response, measured not by the average but by a percentile. The p95 latency is the value that 95% of requests beat. Take 20 requests with latencies in ms: 112, 118, 121, 125, 128, 130, 131, 133, 135, 138, 140, 142, 145, 150, 155, 162, 170, 185, 260, 940. The mean is 186 ms and the median (the mean of the 10th and 11th values) is 139 ms, yet by the nearest-rank method the p95 is the 19th value, 260 ms, and one request took 940 ms. The mean hides the experience of the users in the tail, which is why services have p95 or p99 targets. For text generation, two latencies matter: time-to-first-token (queue, preprocessing, prefill) and the per-token time after that.

Throughput is work per unit time: requests per second, or for language models tokens per second. Training cares about throughput almost exclusively: examples or tokens processed per second per dollar, because no one waits on a single step. Batch inference (scoring a nightly table of ten million emails) also optimizes throughput, since no individual row has a deadline. Online serving needs both, and the two conflict through batch size. Accelerators are most efficient when many sequences share one pass over the weights: reading 2 GB of weights from memory costs about the same whether it serves one sequence or several.

An illustrative decode model: one decoding step takes 20 ms for a batch of one and grows by 2 ms for each additional sequence, step(b) = 20 + 2(b − 1) ms (the numbers are assumed to show the shape, not measured on any system).

for b in (1, 8, 32):
    step_ms = 20 + 2 * (b - 1)               # time for one decode step of the whole batch
    tokens_per_s = b / (step_ms / 1000)      # each sequence emits one token per step
    print(b, step_ms, round(tokens_per_s))
# b=1 : step 20 ms -> 50 tokens/s total,  20 ms per token for the user
# b=8 : step 34 ms -> 235 tokens/s total, 34 ms per token for the user
# b=32: step 82 ms -> 390 tokens/s total, 82 ms per token for the user

The sliders below evaluate the same illustrative model at any batch size, not just 1, 8 and 32.

A continuous-batching serving engine admits new requests into the running batch between steps, which keeps the accelerator busy without holding early arrivals until a batch fills; that scheduling policy, and the trade-offs of queue length versus tail latency, are covered in Model Serving. The right metric follows the job: p95 and time-to-first-token for a chat endpoint, tokens per second per accelerator for a batch scoring job, samples per second and time-to-target-quality for training. No one number describes all three.

Training-serving skew is any difference between how inputs are prepared and distributed during training and during serving. The model is not at fault; it is applying correct weights to inputs it never learned to read. Typical sources are feature code implemented twice (once in a batch pipeline for training, once in the service), a tokenizer or vocabulary version that differs between training and serving, a stale model serving after the data has moved on, and genuine change in the input population.

Here is a plausible incident for the spam classifier. One of its features is sender_age_days, how long the sender's domain has been known. A data-platform team, refactoring the upstream event, changes the field's unit from days to hours and renames nothing; the schema still says int. Training data was exported before the change; the service reads live events after it.

1. 1

   09:00 — upstream deploy

   The event now carries age in hours. Every value is 24 times larger; the types are unchanged, so no validation fails and no error is logged.
2. 2

   09:05 — the model still answers

   A weight learned for "old sender, likely legitimate" now fires on nearly every sender because every value looks old. Mail from new, suspicious domains is scored as trustworthy. Latency and error rate look perfect.
3. 3

   09:20 — input monitoring flags it

   A feature-distribution monitor compares live traffic with the training distribution in four bins. The Population Stability Index is the sum of (live% − train%)·ln(live% / train%). With training shares 40/30/20/10% and live shares 2/5/13/80%, the bin contributions are 1.138, 0.448, 0.030 and 1.456, for a PSI of about 3.07. A common rule of thumb treats values above 0.25 as a major shift; the threshold is a convention, not a theorem.
4. 4

   09:35 — outcome monitoring confirms

   User "not spam"/"report spam" rates, which are delayed labels, start moving, and the share of mail scored above the threshold falls sharply. Because labels arrive late, the input monitor fires first. That is why both exist.
5. 5

   09:50 — remedy and owner

   The first action is mitigation, not retraining: roll the upstream change back or convert at the service boundary. The data platform team owns the schema contract; the ML team owns adding a unit and range check on the feature and a regression test comparing the batch and online implementations on the same raw events. Retraining on hour-valued data is a later, deliberate decision.

The durable lessons are structural: define the feature transformation once and share it between training and serving; pin the tokenizer and feature-schema versions inside the model artifact; validate ranges and units at the service boundary; monitor input distributions (not just latency and errors) with alerts; and compare offline replay of the same raw events through the training and serving paths. For general monitoring of AI applications see AI Observability.

A model in production is the end of a chain, and each link has to be recorded for the chain to be repeatable. A reproducible run pins the dataset version and split, the training code revision, the hyperparameters and random seed, and the library versions, and it records the metrics it produced. Floating-point reductions on parallel hardware can still cause run-to-run differences, so "reproducible" in practice means "explainable and approximately repeatable". The evaluation stage compares the candidate against the current production model on held-out data, including slices that matter (new senders, short emails, each language), not just a single aggregate. Only passing candidates enter the registry as immutable, versioned artifacts bundled with their tokenizer, feature schema and evaluation report.

Canary deployment limits the blast radius of a bad model: if v13 shows worse outcome metrics or drifted inputs on 5% of traffic, the router sends everything back to v12, which is still loaded and unchanged. Rollback is only instant because artifacts are immutable and versioned; "retrain and hope" is not a rollback. Monitoring closes the loop with the retrain decision: retraining is triggered by evidence (drift past a threshold, a drop in outcome metrics, a meaningful volume of new labels, or a fixed schedule), and every retrain must pass the same evaluation gate. Sometimes the right response to a monitoring alert is not retraining at all, as the schema incident shows.

Training and adaptation are mostly one-time (or periodic) expenditures; inference is incurred on every request, for as long as the product lives. Which dominates depends on volume, so reason with a formula and explicit assumptions rather than a universal number:

monthly inference cost = requests × [ (input tokens ÷ 10⁶) × price_in + (output tokens ÷ 10⁶) × price_out ]

requests   = 3_000_000   # per month (assumption)
in_tokens  = 600         # average prompt tokens per request: instructions + context
out_tokens = 250         # average generated tokens per request
price_in   = 0.50        # hypothetical currency units per 1M input tokens
price_out  = 1.50        # hypothetical currency units per 1M output tokens

input_cost  = requests * in_tokens  / 1e6 * price_in    # 3 * 600  * 0.50 = 900.0
output_cost = requests * out_tokens / 1e6 * price_out   # 3 * 250  * 1.50 = 1125.0
print(input_cost + output_cost)                          # 2025.0 per month

# Variant: adaptation lets the prompt shrink to 150 tokens (assumption), same output.
cheaper = requests * 150 / 1e6 * price_in + output_cost  # 225.0 + 1125.0 = 1350.0
saving  = 2025.0 - cheaper                               # 675.0 per month
payback_months = 12_000 / saving                         # one-time adaptation cost (assumption) ~ 17.8

With these assumptions the adapted model saves 675 per month in prompt tokens, so a 12,000 one-time cost pays back in about 17.8 months; if volume were 10× higher the payback would be under two months, and if the prompt did not shrink there would be no saving at all. Input and output tokens are priced differently, output tokens also cost latency because each one needs a decode step, and a longer context raises both cost and KV cache memory, which is why Context Window budgeting is a cost topic as well as a quality one. The comparison is also incomplete without the cost of building a dataset, re-running adaptation for each new base model version, evaluating, and hosting a custom model; broader levers (caching, smaller models, batching) are in Cost Optimization.

Because training and serving fail differently, the first diagnostic question is always which phase the symptom belongs to. Six situations cover most of the ground, and the remedies hardly overlap.

Poor validation performance. Compare training and validation loss. If both are high, the model is underfitting: it needs more capacity, better features, longer training or a lower learning rate. If training loss is low and validation loss is high, it is overfitting or the validation data is drawn from a different distribution: add data, regularize, stop earlier, and check for a leaky or mismatched split. The ML engineer owns this loop, but a data owner is needed when the labels themselves are noisy.

GPU out-of-memory. Covered above: in training reduce micro-batch size or shard state; in serving cap concurrency or context, quantize, or scale out. Owner: the ML platform or infrastructure engineer.

Exploding gradients or a diverging loss. Loss becomes NaN or infinity, as in the η = 0.6 example. Lower the learning rate, add warmup and gradient clipping, check for bad input values or a numerical precision issue (FP16 overflow). This is a training problem and never appears at inference, where no gradients exist. Owner: the ML engineer.

Serving timeout. The model is fine but responses are late: look at the timeline: queue delay from under-provisioning, a long prompt inflating prefill, a long generation, no batching, a cold start from loading weights. Remedies are capacity, limits on input and output length, batching policy, caching and a smaller or quantized model, not retraining. Owner: the serving or SRE team.

Stale model. The model's knowledge or the world's statistics moved. For a classifier, retrain or fine-tune on recent labelled data; for a language model needing current facts, the answer is usually to supply information at request time through retrieval rather than to retrain, as discussed in the Hallucinations lesson. The owner is the product and ML team, together deciding the freshness requirement.

Model drift. A broad term for degradation after deployment, with three distinct causes: the input distribution shifts (data drift), the relationship between inputs and correct output shifts (concept drift: what counts as spam changes), or the pipeline breaks (skew, as in the incident). Input monitors detect the first and third; only outcome metrics and fresh labels reveal the second. Treat the cause before choosing retraining.

This chapter treated the model as a function with fixed numbers inside it, and examined what surrounds learning and serving it. For a language model the function begins by converting text into integer units and ends by producing a probability for the next one, and the cost, memory and latency figures above all trace back to the number of those units: the 600 input and 250 output tokens in the cost example, the 4,096 tokens in the KV cache, the 63 decode steps in the timeline. Before going further it is worth settling what a token is, since every sequence length, price, and limit that follows is denominated in them. The next lesson starts there, and Transformer Basics later opens the forward pass that was only drawn as a box here.

1. 1

   Training adjusts weights and needs gradients and optimizer state (roughly 16 bytes per parameter under the page's mixed-precision accounting, versus 2 bytes for FP16 weights alone); inference only runs the forward pass, and its growing memory term is the KV cache.

2. 2

   Training is judged on throughput and cost per run; online inference is judged on latency, and the two pull against each other through batching: a bigger batch raises tokens per second but lengthens each step.

3. 3

   Training loss alone proves nothing. Watch validation loss and guard against leakage such as near-duplicates across the split; a rising validation loss while training loss falls signals overfitting.

4. 4

   Prompting, including few-shot examples, is inference and changes no weights; fine-tuning continues training from pretrained weights, and its one-time cost only pays back if it cuts recurring per-request cost enough.

5. 5

   Define the feature transformation once and share it between training and serving, and pin tokenizer and vocabulary versions; otherwise correct weights receive inputs they never learned to read.

6. 6

   Diagnose by phase first: poor validation performance, exploding loss and training OOM belong to training; timeouts, serving OOM and stale or drifting models belong to serving. Canary a new model on a small share of traffic before full rollout.