Transformer Basics

Watch the shift from sequential recurrence to global token interaction and see why that changed the scale of modern language modeling.

A transformer model is a neural network whose architecture has three parts: token input (text split into tokens, then embedded and given position information), a stack of attention plus MLP blocks, and an output mechanism (a head that maps the final vectors to logits over the vocabulary or to another task output). It turns a sequence of tokens into a sequence of vectors, one per position, by repeating a block in which every position first gathers information from other positions (self-attention) and then transforms what it gathered on its own (a feed-forward network). The path is: text → tokenizer → integer IDs → token embeddings → added position information → a stack of identical-shaped blocks → final hidden states → a head that maps them to whatever the task needs, such as logits over the vocabulary for the next token. Everything in the stack preserves the shape [batch, sequence, d_model], which is what makes the design stackable and parallelisable.

"Transformer" names a family, not one fixed stack. Encoder-only, decoder-only and encoder–decoder variants differ in which positions may read which, and in what the model is trained to predict. This chapter follows the whole path once at the level of shapes and responsibilities. The query/key/value arithmetic inside the attention step is deliberately left to Attention Mechanism. The two input stages — turning text into IDs and IDs into vectors — are covered in Tokens Explained and Embeddings Explained.

Take the sentence "The animal did not cross the road because it was tired." A reader resolves "it" to "the animal" because roads do not get tired. Change the last word to "wide" and "it" now means the road. The word "it" is identical in both sentences; its meaning is supplied by words elsewhere. Any model of language therefore needs a way for the representation of one position to depend on other positions, sometimes far away.

Processing each word in isolation, as a lookup table would, cannot do this: the vector for "it" would be the same in every sentence. A bag of words that merely counts neighbours loses order, and order matters ("the customer refunded the store" is not "the store refunded the customer"). The model needs a mechanism that is context-dependent, order-aware and able to link distant positions without the link degrading with distance.

Before 2017 the dominant sequence models were recurrent networks (RNNs, and their gated variants such as LSTMs). A recurrent network reads token 1, updates a fixed-size hidden state, reads token 2, updates the state again, and so on. Two consequences follow. First, training cannot process positions of one sequence simultaneously, because the state after token t needs the state after token t − 1; on hardware built for large parallel matrix multiplies this wastes capacity. Second, everything the model knows about the past must be squeezed through that one state, and information from early tokens must survive many updates to influence later ones. Gating eases this but does not remove the bottleneck.

Attention was first added to recurrent encoder–decoder translation models to let the decoder look back at all encoder states instead of one summary. The transformer paper (Vaswani et al., "Attention Is All You Need", arXiv:1706.03762) went further: drop recurrence and build the network from attention plus position-wise feed-forward layers. Every position can read every other (permitted) position in a single layer, so the path between two tokens is one step long, and all positions of a training sequence are computed in one batch of matrix multiplications.

The diagram shows the path for a decoder language model. Variants are noted where they differ. Three symbols recur throughout: n is the sequence length in tokens, d_model is the width of every hidden vector, and V is the vocabulary size.

Dimensions are the glue. The tokenizer outputs n integers. The embedding table has shape [V, d_model]; looking up each ID gives [n, d_model]. Position information is added to (or, in rotary schemes, applied inside) that tensor without changing its shape. Each block maps [n, d_model] to [n, d_model]; this uniformity is why depth can be scaled by simply stacking more blocks. The output head multiplies by a matrix of shape [d_model, V], producing [n, V] logits. Some models reuse the embedding table, transposed, as that matrix ("weight tying"); others learn it separately.

To make the shapes concrete, run ["The", "refund", "arrived"] through a toy decoder with batch = 1, n = 3, d_model = 8, a hypothetical vocabulary of V = 32,000, and illustrative IDs. These are shapes and sizes, not outputs from a real model; a real tokenizer might split the words differently and real d_model values are far larger.

1. 1

   Tokenize

   Text becomes IDs of shape [1, 3], for example [17, 402, 93]. IDs are arbitrary table indices; 402 is not "larger" than 17 in any meaningful way.
2. 2

   Embed

   Each ID selects a row of the [32000, 8] table. Result: [1, 3, 8]. The table holds 32,000 × 8 = 256,000 learned numbers.
3. 3

   Add position

   A position signal of shape [3, 8] is added (broadcast over the batch), so "refund" at position 1 differs from "refund" at position 5. Shape stays [1, 3, 8].
4. 4

   Attention in each block

   Every position produces a mixture of information from permitted positions. Output [1, 3, 8]. In a decoder, position 0 sees only itself, position 1 sees 0–1, position 2 sees 0–2.
5. 5

   MLP in each block

   Each of the three rows is transformed independently by the same small network. Output [1, 3, 8]. Repeat attention + MLP N times.
6. 6

   Output head

   Each row is projected to V = 32,000 scores: [1, 3, 32000] — 96,000 numbers. Row 0 scores what could follow "The", row 1 what could follow "The refund", row 2 what could follow "The refund arrived".

Notice that a decoder produces n predictions in one pass: it has a next-token distribution for every prefix. During training all of them are scored against the actual next tokens at once. During generation only the last row matters, because the earlier rows predict tokens we already have.

Self-attention on its own treats its input as an unordered collection. If you permute the input rows and apply attention with no mask and no position signal, the output rows are permuted in the same way (the layer is permutation-equivariant): the representation of "refund" does not depend on whether it came first or last, only on which other tokens are present. Word order is meaning, so position has to be injected. There are three common families, and different models choose differently.

Learned absolute position embeddings

A second table of shape [max_positions, d_model] is learned along with the rest of the model; row p is added to the token embedding at position p. It is simple and flexible, but the model only has trained vectors for positions up to max_positions, so longer inputs are out of range.

Sinusoidal encoding

The original paper used fixed, unlearned vectors. For position pos and dimension pair i (0 ≤ i < d_model/2):

PE(pos, 2i)   = sin( pos / 10000^(2i / d_model) )
PE(pos, 2i+1) = cos( pos / 10000^(2i / d_model) )

For d_model = 8 the four pairs use divisors 10000⁰ = 1, 10000^0.25 = 10, 10000^0.5 = 100 and 10000^0.75 = 1000. Position 0 gives [0, 1, 0, 1, 0, 1, 0, 1]. Position 1 gives [sin 1, cos 1, sin 0.1, cos 0.1, sin 0.01, cos 0.01, sin 0.001, cos 0.001] ≈ [0.841, 0.540, 0.100, 0.995, 0.010, 1.000, 0.001, 1.000], and position 2 gives ≈ [0.909, −0.416, 0.199, 0.980, 0.020, 1.000, 0.002, 1.000]. The fast pair changes visibly from one position to the next; the slow pairs barely move, which lets the model distinguish nearby and far-apart positions from different dimensions. The paper's authors hypothesised that this form makes relative offsets easy to express, because the encoding at pos + k is a linear function of the encoding at pos.

Rotary position embeddings (RoPE) and relatives

Many recent decoder models do not add a position vector to the input at all. Rotary embeddings instead rotate pairs of dimensions in the query and key vectors by an angle proportional to position (RoFormer, Su et al., arXiv:2104.09864), so that the attention score between two tokens depends on the difference of their positions. Other schemes add a position-dependent bias directly to attention scores. These choices matter in practice: they influence how well a model behaves on inputs longer than those it was trained on. No single method is universal, so a statement such as "transformers add sinusoids to the embeddings" describes one design, not the family.

A block has two sub-layers, each wrapped in a residual connection and a normalisation step. The first sub-layer is multi-head self-attention: several attention computations run in parallel, each on a smaller slice of the width (with h heads, each head typically has d_head = d_model / h), and their outputs are concatenated and mixed by an output projection back to d_model. Separate heads can specialise in different relationships, although no head is guaranteed to have a clean, human-readable role. The second sub-layer is a position-wise feed-forward network (the MLP).

In symbols, a pre-norm block computes h₁ = x + Attn(Norm(x)) and y = h₁ + MLP(Norm(h₁)). The original paper used the post-norm ordering, Norm(x + Attn(x)), and many later models switched to pre-norm because it tends to train more stably at depth; some use other normalisation functions (RMSNorm instead of LayerNorm) or extra norms around sub-layers. The ordering is a design decision that varies by model, not a defining property.

What the two sub-layers contribute

Attention is the only place in the block where information moves between positions. For a given position it forms a weighted combination of value vectors computed from the positions it is allowed to read; the weights come from learned query/key comparisons. The full computation, with real numbers, is worked through in Attention Mechanism. The MLP does the complementary work: it takes each position's vector alone, expands it to a wider hidden layer (commonly about four times d_model, or a comparable size in gated variants), applies a nonlinearity, and projects back. It mixes the features within one position and never looks at another row. Without it, a stack of attention layers would mostly be repeated weighted averaging of linear projections; the nonlinearity and extra capacity are what let each position compute new features from what attention gathered. A gated MLP (for example SwiGLU-style layers) multiplies two parallel projections elementwise; it is common in recent models but remains a variant.

Residual connections give each sub-layer the job of computing a correction to the running representation instead of rebuilding it. They provide a direct path for the signal and gradients through dozens of layers, which is essential for training deep stacks. Normalisation rescales each position's vector to a controlled magnitude so that activations do not drift in scale from layer to layer. Both are training-stability tools rather than sources of linguistic knowledge. Dropout and bias terms vary by implementation.

The same block can be assembled in three ways, distinguished by which positions each position may read. An encoder–decoder model reads the whole input with an encoder, then generates the output with a decoder that consults the encoder's states through cross-attention.

Encoder-only models use bidirectional self-attention: every token reads every other token, before and after it. The result is a contextual vector per token, which is the right shape for understanding tasks. Example requests: classify a support message as "billing" versus "shipping"; tag which spans of a message are order numbers; produce a sentence vector for search (see Embeddings Explained). The model does not generate free text on its own; a small head sits on top of the final hidden states.

Decoder-only models add a causal mask: position i may read positions 0…i and nothing later. The mask is what makes next-token training honest: row i predicts token i + 1, and if it could read that token it would simply copy it. It also makes generation possible, because a prefix can be extended without invalidating the work done for earlier positions. Example request: "Draft a reply explaining that the refund takes five to seven business days." This is the layout used by most current chat LLMs; the system around the model (prompt templates, tools, safety layers) is a separate matter covered in the LLM Overview.

Encoder–decoder models run an encoder over the input and a causal decoder over the output produced so far; each decoder block has an extra cross-attention sub-layer whose queries come from the decoder and whose keys and values come from the encoder's final states. The input is read bidirectionally and once; the output is generated left to right while consulting it. Example request: translate a French refund notice into English, or summarise a long ticket thread into two sentences.

These categories are a map, not a law. Decoder-only models can classify, extract and translate when asked in a prompt; encoders can be adapted into generators in specialised setups; and some architectures use prefix-style masks where the prompt is read bidirectionally while the continuation is causal. The mask and the training objective together define the behaviour.

What a transformer learns depends on what loss it is trained against. For a decoder language model the objective is next-token prediction: at each position, the cross-entropy loss compares the vocabulary distribution (softmax of the logits) with the actual next token, averaged over all positions. Prompting such a model later never changes its weights; weight updates happen only during training or fine-tuning (see Training vs Inference).

Encoder-only models are often pre-trained by masking a fraction of the input tokens and asking the model to recover them from the surrounding context in both directions, then fine-tuned with a small head for classification or tagging. Encoder–decoder models are trained on conditional generation: given the input sequence, predict the target sequence token by token. Architecture and objective are therefore related but separable: a bidirectional encoder cannot be trained with plain next-token loss without leaking the answer, and a causal decoder trained only with next-token loss will be better at continuation than at filling a gap in the middle. Later stages such as instruction tuning and preference alignment change behaviour without changing the architecture; see Next-Token Prediction and Model Alignment.

A decoder answering a 12-token prompt works in two phases. In prefill the whole prompt is processed in one forward pass: all 12 positions are computed in parallel, and as a by-product each layer's key and value vectors for those positions are saved. The final row of logits gives the distribution for the first generated token. In decode, a token is chosen (greedy or sampled; see Temperature), appended, and a forward pass is run for just that one new position, which reads the saved keys and values of all earlier positions and adds its own. Repeat until a stop condition.

Without caching, step t would recompute keys and values for all t − 1 earlier positions in every layer. The KV cache stores them, so each decode step computes projections only for the new token and then attends over the stored ones. That turns per-step work from reprocessing the whole sequence into roughly linear-in-length attention reads plus the fixed per-token cost of the weights. It does not remove the step-by-step dependency, and the cache itself uses memory (see the sizing below). Decode steps are typically limited by how fast weights and cache can be read from memory rather than by arithmetic, which is why batching many requests, paging the cache and similar techniques matter. Those serving concerns belong to Inference Infrastructure and Model Serving; the point here is that the architecture determines the shape of the problem: wide parallel prefill, narrow sequential decode.

Step through a 12-token prompt and three generated tokens to see what the KV cache saves at each step.

In a straightforward implementation, each head forms an n × n matrix of scores, one entry per (query position, key position) pair, in every layer. Work and memory for that matrix grow with n²: doubling the context from 4,096 to 8,192 tokens multiplies the score entries by (8192/4096)² = 4.

Concretely, for n = 4096 a single head holds 4096² = 16,777,216 scores per layer, and with 32 heads that is 536,870,912 entries — about 1.0 GiB at 2 bytes each, if the full matrix were stored. Real implementations often avoid materialising it: fused attention kernels compute the softmax in tiles and discard scores as they go, so memory for scores can stay far below that figure while the arithmetic remains quadratic. Other designs change the pattern itself — sliding windows, sparse or grouped patterns, linear-attention approximations — trading generality for cost. The MLP and projections, by contrast, scale linearly with n. Which term dominates depends on the ratio of n to d_model and on the implementation.

Parameters, activations and cache are different budgets

Three quantities are routinely confused. Parameters are the learned weights: fixed once training ends, independent of the prompt. Activations are temporary tensors computed during a forward pass. The KV cache is per-request state that grows with every token kept in context. Using illustrative values V = 50,000, d_model = 4096, L = 32 layers, h = 32 heads of d_head = 128:

Embedding table      = V x d_model                  = 50,000 x 4,096  = 204,800,000 parameters
Per-layer weights    ~ 4*d^2 (attention Q,K,V,O)
                     + 8*d^2 (MLP, 4x expansion, two matrices)
                     = 12 * 4096^2                  = 201,326,592 parameters
All blocks (L = 32)  = 32 * 201,326,592             = 6,442,450,944  (~6.4 billion)
(Biases, norm scales, separate output head omitted.)

KV cache per token   = 2 (K and V) * L * d_model * bytes
                     = 2 * 32 * 4096 * 2 B (fp16)   = 524,288 B = 0.5 MiB
KV cache, n = 4096   = 4096 * 0.5 MiB               = 2.0 GiB  per sequence
If only 8 shared KV heads of 128 dims (grouped-query attention): 0.5 GiB per sequence

Two readings follow. First, parameter count sets the baseline memory needed to hold the model, but it says nothing about how many users or how long a context can be served: those depend on the KV cache, which multiplies by the number of concurrent sequences. Second, architectural choices such as grouped-query attention directly shrink serving memory without changing d_model. See Context Window for what a context limit means to an application, and Cost Optimization for the economics.

Move the context length to see the same sizing assumptions as the block above recomputed.

A transformer produces a learned representation, and a decoder turns it into a probability distribution over the next token. Neither is a database lookup. A fluent continuation can state a refund policy that was never written down, because the objective rewards plausible text, not verified text; mitigating that is an application concern (retrieval, citations, checks) discussed under Hallucinations and in RAG.

Several limits come directly from the mechanics above. Quality can degrade on long inputs: a model may handle a position range poorly if it saw few examples of that length in training, and relevant facts buried in the middle of a long prompt are not always used as reliably as those at the ends. The position scheme bounds how far a model extrapolates beyond its training lengths. Outputs are sensitive to prompt wording, because the same tokens in a different order are a different input. Compute and memory grow with context, as sized above. And attention weights, although easy to plot, are not a faithful explanation: a high weight shows where information was read from in one head of one layer, not why the model produced an answer, and information is mixed across many heads and layers. Treat attention maps as a debugging hint, not as evidence of reasoning.

Suppose the online store needs three features. Routing each incoming message to billing, shipping or returns is a classification over a short text: an encoder-only model with a classification head is small, fast and cheap to run on every message, and bidirectional context helps. Finding similar past tickets needs a vector per message; an encoder trained for that purpose produces it in one forward pass. Drafting a customer-facing reply is open generation, where a decoder-only model is the natural choice. Translating or summarising a long thread into a fixed output is a fit for encoder–decoder, which reads the input once with full bidirectional context.

Trade-offs cut across these defaults. A large decoder can do all three jobs through prompting, which reduces the number of models to operate but costs more per request and is less predictable for narrow classification than a small tuned encoder. Encoder-only models do not produce text, and their input length is typically limited. Encoder–decoder models require running two stacks. The right choice follows from the task's output shape, latency budget and the volume of requests, and then should be validated on your own data rather than assumed from architecture labels.

The sketch below is framework-neutral pseudocode and not runnable as written. It shows shapes, residuals, the mask and the output head. It omits dropout, biases, grouped-query attention, rotary application, caching and numerical details, and its helper functions (matmul, softmax, norm,gelu) are assumed.

# x: [batch, n, d_model]   h: number of heads   d_head = d_model / h

def causal_mask(n):                       # [n, n]; True where reading is allowed
    return [[j <= i for j in range(n)] for i in range(n)]

def self_attention(x, W_q, W_k, W_v, W_o, mask):
    q, k, v = x @ W_q, x @ W_k, x @ W_v           # each [batch, n, d_model]
    q, k, v = split_heads(q), split_heads(k), split_heads(v)   # [batch, h, n, d_head]
    scores = (q @ k.transpose(-1, -2)) / sqrt(d_head)          # [batch, h, n, n]
    scores = where(mask, scores, -inf)                         # block future positions
    weights = softmax(scores, axis=-1)                         # rows sum to 1
    out = merge_heads(weights @ v)                             # [batch, n, d_model]
    return out @ W_o                                           # [batch, n, d_model]

def mlp(x, W_1, W_2):                      # W_1: [d_model, 4*d_model], W_2: [4*d_model, d_model]
    return gelu(x @ W_1) @ W_2             # applied to every position independently

def block(x, p, mask):                     # pre-norm variant
    x = x + self_attention(norm(x, p.n1), p.W_q, p.W_k, p.W_v, p.W_o, mask)
    x = x + mlp(norm(x, p.n2), p.W_1, p.W_2)
    return x                                                   # [batch, n, d_model]

def decoder_lm(ids, params):               # ids: [batch, n] integers
    n = ids.shape[1]
    x = params.embed[ids] + params.pos[:n]                     # [batch, n, d_model]
    mask = causal_mask(n)
    for layer in params.layers:            # N identical-shaped blocks
        x = block(x, layer, mask)
    x = norm(x, params.final_norm)
    return x @ params.W_out                # logits [batch, n, V]

Reading the code against the earlier sections: the embedding lookup and position addition are the first two lines of decoder_lm; block is the two-residual structure; the mask is applied to the scores before the softmax so that blocked positions receive exactly zero weight; and the final line is the output head. Swapping causal_mask for an all-true mask yields encoder-style self-attention, and adding a second attention call whose keys and values come from encoder states yields an encoder–decoder block.

The architecture is now a map: tokens become vectors, blocks move information between positions and refine it within each position, and a head converts the result into the task's output. Two questions remain open at this level of detail. How exactly does a position decide how much to read from each other position — what are queries, keys and values, and what does the softmax do with them? And how does a decoder-only stack, given those logits, actually choose and emit text? The first is the subject of the next lesson, with a small attention matrix computed by hand. The second is taken up in the LLM track, starting with Next-Token Prediction.

1. 1

   Attention is the only place in a block where information moves between positions; the feed-forward sublayer, normalisation and residuals act on each position independently.

2. 2

   Dimensions are the glue: n token IDs become an [n, d_model] matrix and every block preserves that shape, which is what lets blocks be stacked.

3. 3

   Attention alone is order-blind, so position must be injected (learned, sinusoidal, rotary or attention-bias schemes); no single method is universal.

4. 4

   Encoder-only, decoder-only and encoder–decoder differ in which positions may read which. The causal mask is what makes next-token training honest, but the architecture is not the same as the training objective.

5. 5

   Generation has two phases: prefill processes the whole prompt in one pass, and decode produces one token at a time with a KV cache that trades memory for avoided recomputation.

6. 6

   Parameters, activations and KV cache scale differently. Parameter count sets baseline memory but not how many users or how long a context fit, and n² attention scores make long contexts expensive.

7. 7

   A transformer yields a learned probability distribution, not a database lookup, so it does not guarantee factual accuracy or reliability on very long inputs.