Tokens Explained

Watch raw text break into tokens, map to IDs, enter the model, and come back out as generated text.

Tokenization is how a large language model turns text into the units it actually reads and writes. A tokenizer splits text into tokens, pieces drawn from a fixed vocabulary, and maps each piece to an integer token ID; the model never sees letters or words, only those integers. Tokens need not equal words: a piece can be a whole common word, a fragment of a rarer one, a punctuation mark, a run of whitespace, or even a fragment of one character's bytes.

For example, a tokenizer might split "unrefundable!" into the pieces un, refund, able, !, which it then looks up as four IDs. This split is illustrative: the actual pieces and ID values depend on the tokenizer, which is why the same sentence can cost different amounts of context in different models. The worked example below uses byte-pair encoding (BPE), a widely used tokenizer family chosen because its mechanism is easy to follow by hand, not because it is the only algorithm.

Tokens are also the unit of accounting. A model's context window, the length of what it can generate, the time it takes to process a prompt, and what most hosted APIs charge are all counted in tokens, not in characters or words. This chapter follows text through the tokenizer, builds a miniature tokenizer by hand so that the mechanism is visible, shows what the IDs do and do not mean, and then turns to the engineering that depends on them: budgets, cost, truncation, and measuring honestly. Where the tokens go next, into learned vectors, is the subject of Embeddings Explained.

In natural language processing (NLP), tokenization is the step that splits raw text into units a program can count and look up. Classic NLP often split on words; modern language models use subword pieces instead, because a fixed vocabulary of subwords stays a manageable size and still covers rare words, as the section on choosing what a piece should be explains. The pieces are then mapped to integer IDs, the same pipeline shown in the figure under From Text to Embedding Rows.

Imagine the online store's support assistant receives five messages, and the engineer who owns its budget wants to know how long each one is. A short, common word such as refund is probably a single piece in almost any modern vocabulary. A longer technical word such as refundability is probably built from a few pieces. A German compound such as Erstattungsanspruch (roughly, "entitlement to reimbursement") has a length in characters that looks similar to the English word but may break into several pieces, because the tokenizer saw less of that string when its vocabulary was built. A line of code carries punctuation and operators, each of which can be its own piece. An emoji is a single visible symbol but several bytes underneath.

Read the figure for the pattern rather than the counts. Nothing in it was produced by a commercial tokenizer; the splits are invented to show typical behaviour. The reasons behind that behaviour are mechanical, and they are what the rest of the chapter explains: a vocabulary has a fixed size, it was built by counting what appeared often in some training text, and anything that was rare in that text is assembled from smaller parts. A consequence worth stating now is that a "word count" or "character count" can never reliably predict token count. The only reliable way to know is to run the exact tokenizer that the target model uses.

The path from a string to something a neural network can compute with has four stages, and the first three belong to the tokenizer, which is ordinary deterministic software, not a neural network.

Normalisation and pre-tokenization

Before any pieces are found, many tokenizers apply a normalisation step whose details vary by design: some convert certain Unicode sequences to a canonical form, some lowercase, many do nothing at all so that the original text can be recovered exactly. Then a pre-tokenization rule typically cuts the string into coarse chunks, often along whitespace and punctuation boundaries, and subword splitting happens inside each chunk. Two details of this stage surprise people. Whitespace is frequently part of a piece, so " refund" (with a leading space) and "refund" can be different tokens with different IDs. And case matters in tokenizers that do not lowercase, so Refund and refund need not be the same piece. None of these rules is universal; they are decisions made by whoever designed that tokenizer.

The vocabulary: pieces and IDs

The vocabulary is a table that maps every allowed piece to an integer between 0 and V − 1, where V is the vocabulary size (a design choice, commonly in the tens of thousands up to a few hundred thousand, depending on the tokenizer). Encoding finds the pieces and emits their IDs; decoding does the reverse and concatenates the text of each piece. For a well-formed tokenizer, decoding the encoded IDs reproduces the original string.

Why an ID is a lookup index and not a meaning

Take the pieces ['un', 'believ', 'able'] and suppose this tokenizer assigns the invented IDs 3841, 17206 and 912. These numbers are positions in a list, nothing more. An ID that is numerically close to another says nothing about meaning: ID 912 could be a suffix and ID 913 could be a Cyrillic letter, because the order is an accident of how the vocabulary was assembled. The same string could be a single piece, unbelievable, with ID 55021 in another model's vocabulary, and the number 3841 would refer to something entirely different there. IDs from one tokenizer are therefore meaningless to any other model, and arithmetic on IDs is meaningless within a model too.

What gives a token any numerical meaning is the next step. The model owns an embedding table E with V rows and d columns (d is the model's hidden width). An ID is used as a row index. Using the same toy convention with d = 4 so that the arithmetic fits on a line:

E[3841]  = [ 0.12, -0.40,  0.07,  0.55]    # row for "un"
E[17206] = [-0.31,  0.22,  0.64, -0.08]    # row for "believ"
E[912]   = [ 0.05,  0.18, -0.27,  0.33]    # row for "able"

X = stack of the three rows            # shape [3, 4] = [sequence length n, width d]

The matrix X, with one row per token, is what the transformer actually processes. The table E is a learned parameter: during model training its values are adjusted so that rows become useful for predicting text, which is why two tokens used in similar contexts end up with similar rows, a topic that Embeddings Explained develops. As a sizing example (units: values, then bytes), a table with V = 32,000 and d = 4,096 holds 32,000 x 4,096 = 131,072,000 values, which at 2 bytes each is 262,144,000 bytes, or exactly 250 MiB (about 262 MB in decimal units). Those figures are an illustration of how vocabulary size trades against memory; they are not the configuration of any particular model.

There are three obvious ways to cut text into units, and each fails in a way that explains why modern systems settle in the middle.

Characters give a tiny vocabulary (a few hundred symbols can cover a lot of text) and never meet an unknown input, but sequences become long. Every extra position costs memory and attention computation, and the model must learn from scratch that the letters r, e, f, u, n, d together mean something. Whole words give short sequences and meaningful units, but the vocabulary must contain every form of every word in every language, so it explodes, and any word not in it (a new product name, a typo, an inflection) becomes a single "unknown" placeholder that discards the information. Subwords sit between: frequent strings get their own pieces, rare strings are composed from smaller pieces, and the vocabulary stays a fixed manageable size.

Byte-level approaches

A further design decision is what the base alphabet is. If the smallest unit is a Unicode character, there are well over a hundred thousand possible base symbols, and a rare one may still be unknown. If the smallest unit is a byte of the UTF-8 encoding, the base alphabet is exactly 256 values and every possible string can be represented, because every string is bytes. Byte-level tokenizers therefore have no unknown-text failure; the price is that a rare character can cost several tokens, since each byte may end up as its own piece. This is why emoji and text in scripts that were rare in the tokenizer's training data can be disproportionately long, and why a streaming client may receive half of a character (the section on generation returns to this).

BPE, WordPiece and Unigram at a conceptual level

Three algorithms dominate how a subword vocabulary is built. Byte-pair encoding (BPE), adapted to subword segmentation for neural translation by Sennrich, Haddow and Birch (arXiv:1508.07909), starts from base symbols and repeatedly merges the most frequent adjacent pair into a new symbol; the next section does this by hand. WordPiece also grows a vocabulary by merging, but chooses merges by a likelihood-based score rather than raw pair counts, and marks word-internal pieces with a prefix such as ## in common implementations. Unigram works in the opposite direction: it starts with a large candidate set of pieces, assigns each a probability, and repeatedly prunes the pieces whose removal hurts the overall likelihood least; at encode time it picks the most probable segmentation, which means one string can in principle have several valid segmentations. All three yield the same kind of artifact, a vocabulary plus rules, which is why the rest of the chapter applies to any of them.

Training a tokenizer is not training a model

These algorithms run once, over a text sample, before the language model exists. They count and merge or prune; there is no loss function, no gradient and no neural network. The resulting tokenizer is then frozen and the language model is trained on text encoded by it, which is the gradient-based process described in Training vs Inference. Two consequences follow. The tokenizer reflects whatever text it was built from, so languages and domains that were sparse in that sample tend to be segmented into more, smaller pieces. And the pairing is rigid: a model trained with one tokenizer cannot be given IDs from another, because row 3841 of its embedding table was trained to mean what the original tokenizer's piece 3841 meant.

The algorithm is simple enough to execute on paper, and doing so makes every later statement about vocabularies concrete. Everything in this section is a deliberately tiny, illustrative example; real tokenizers learn thousands of merges from enormous text samples and use byte-level base symbols, regular-expression pre-tokenization and other details that are omitted here.

The corpus and the starting symbols

Suppose the training sample, after splitting into words, contains low five times, lower twice, new six times and newer three times. We mark the end of each word with the symbol _ so the algorithm can distinguish a word-final w from a word-internal one. The starting segmentation is one symbol per character:

5 x  l o w _
2 x  l o w e r _
6 x  n e w _
3 x  n e w e r _

base vocabulary: l o w e r n _         (7 symbols)
total symbols in corpus: 5*4 + 2*6 + 6*4 + 3*6 = 74

Each round the algorithm counts every adjacent pair of symbols, weighting each occurrence by the word's frequency, picks the most frequent pair, replaces every occurrence with one new symbol, and records the merge. In round 1 the counts are: (w,_) = 5 + 6 = 11, (n,e) = 6 + 3 = 9, (e,w) = 6 + 3 = 9, (l,o) = 5 + 2 = 7, (o,w) = 5 + 2 = 7, and several pairs at 5 or below. The winner is (w,_). We break ties alphabetically; any fixed rule works, and real implementations define their own.

merge  pair chosen   count   new symbol   corpus after the merge
  1    (w, _)         11     w_           5 x l o w_ | 2 x l o w e r _ | 6 x n e w_ | 3 x n e w e r _
  2    (n, e)          9     ne           ... 6 x ne w_ | 3 x ne w e r _
  3    (l, o)          7     lo           5 x lo w_ | 2 x lo w e r _
  4    (ne, w_)        6     new_         6 x new_
  5    (e, r)          5     er           2 x lo w er _ | 3 x ne w er _
  6    (er, _)         5     er_          2 x lo w er_ | 3 x ne w er_

vocabulary: 7 base symbols + 6 merged = 13 symbols
total symbols in corpus after merge 6: 31   (was 74)

Watch what the merges do. Frequent whole words (new_) become single symbols; frequent fragments (lo, er_) become reusable parts; and the corpus shrinks from 74 symbols to 31, which is exactly what a tokenizer is for. The ordered list of merges is the learned artifact: it is the tokenizer. In this example the sequence of corpus lengths after each merge is 63, 54, 47, 41, 36, 31.

Step through the six merges below; the table is produced by running the same algorithm on the same corpus.

Encoding a string the tokenizer has never seen

To tokenize new text, split it into base symbols and replay the recorded merges in the order they were learned. The string lower becomes lo w er_, three pieces. The unseen word renew begins as r e n e w _. Merge 1 joins w_, giving r e n e w_; merge 2 joins the adjacent n e into ne; merge 4 joins ne w_ into new_. The script below reports ['r', 'e', 'new_']: the unseen word is assembled from two single letters and a known piece. The word slow contains s, a symbol this toy vocabulary never saw. Our toy encoder simply carries it through as a raw character, giving ['s', 'lo', 'w_']; a real tokenizer uses the 256 byte values as its base alphabet precisely so that no input can ever be unrepresentable.

from collections import Counter

corpus = {"low": 5, "lower": 2, "new": 6, "newer": 3}   # word -> frequency
EOW = "_"                                                 # end-of-word marker
words = {tuple(w) + (EOW,): f for w, f in corpus.items()}

def pair_counts(words):
    counts = Counter()
    for symbols, freq in words.items():
        for pair in zip(symbols, symbols[1:]):
            counts[pair] += freq
    return counts

def apply_merge(symbols, pair):
    out, i = [], 0
    while i < len(symbols):
        if i + 1 < len(symbols) and (symbols[i], symbols[i + 1]) == pair:
            out.append(symbols[i] + symbols[i + 1]); i += 2
        else:
            out.append(symbols[i]); i += 1
    return tuple(out)

merges = []
for step in range(1, 7):
    counts = pair_counts(words)
    best = min(counts, key=lambda p: (-counts[p], p))     # highest count, ties: alphabetical
    merges.append(best)
    words = {apply_merge(s, best): f for s, f in words.items()}
    print(step, best, counts[best], [" ".join(s) for s in words])

def encode(word):
    symbols = tuple(word) + (EOW,)
    for pair in merges:                                   # replay merges in learned order
        symbols = apply_merge(symbols, pair)
    return list(symbols)

for w in ["lower", "newest", "slow", "renew"]:
    print(w, encode(w))

Note what the example shows about quality: newest never saw est in training, so it falls apart into five pieces, whereas the training word new is one piece. The same effect, at a vastly larger scale, is why strings that were common in a tokenizer's training text are cheap and strings that were not are expensive.

A vocabulary usually contains entries that never come from ordinary text. Models are trained with markers such as a beginning-of-sequence token, an end-of-sequence token (the model emitting it is how generation signals "I am done"), a padding token used to make sequences in a batch the same length, separators between segments, and, for chat models, markers that delimit the role of each message. Some models also reserve tokens for tool calls or structured output. Which of these exist, what they are called and how they are written is entirely model- and provider-specific; none of them is universal, and code should never assume a particular marker.

A chat request is therefore not sent to the model as the plain text of the conversation. The serving layer applies a template that wraps each message in role markers, and the model sees the wrapped result. With an invented template (not any vendor's real format) the store's first turn might look like this:

<|bos|>
<|system|>You are a support assistant for an online store. Cite policy IDs.<|end|>
<|user|>My order was charged twice; when will the refund arrive?<|end|>
<|assistant|>

Two practical lessons follow. First, these tokens consume context and are normally billed as input, so a conversation made of many very short messages pays proportionally more framing overhead than the visible text suggests; the exact overhead is a property of the model's template and must be measured. Second, the final marker, the one that opens the assistant's turn, is how a chat model knows it is its turn to speak. Special tokens are meant to be inserted by the template, not by users. Many tokenizer libraries therefore either refuse or escape special-token strings that appear inside ordinary text by default, and you should check what yours does before passing untrusted input through it: a user typing the literal characters of a marker should not be able to forge a role boundary.

Because vocabularies reflect their training text, efficiency is uneven across kinds of input. Text in a language that was well represented in the tokenizer's training sample typically compresses well: most words are one or two pieces. Text in a language or script that was thinly represented splits into smaller pieces, sometimes down to individual bytes, so the same content costs more tokens, more context and more money. This is a property of how a particular tokenizer was built, not of the language, and different tokenizers treat the same language very differently.

The byte level makes the underlying arithmetic visible. UTF-8 uses a variable number of bytes per character, and those byte counts are fixed facts you can verify yourself:

for s in ["refund", "é", "返金", "日本語", "😀"]:
    print(repr(s), "characters:", len(s), "UTF-8 bytes:", len(s.encode("utf-8")))
# 'refund' characters: 6 UTF-8 bytes: 6
# 'é'      characters: 1 UTF-8 bytes: 2
# '返金'   characters: 2 UTF-8 bytes: 6
# '日本語' characters: 3 UTF-8 bytes: 9
# '😀'     characters: 1 UTF-8 bytes: 4

That upper bound is the useful design fact: a byte-level tokenizer never needs more tokens than bytes, and the tokenizer's merges decide how far below that bound a given string lands. An accented letter composed as one code point (é, 2 bytes) and the same letter composed as e plus a combining accent (3 bytes, 2 characters) look identical on screen and are different strings; unless the tokenizer normalises them, they can tokenize differently, which is one of the reasons text from different sources should be normalised before counting or comparing.

Code, whitespace and numbers

Source code is dominated by punctuation, identifiers and indentation. Tokenizers built with code in their training text often have dedicated pieces for common runs of spaces, for keywords and for frequent operators; tokenizers without such exposure spend several pieces on the same line. Indentation therefore has a direct cost: pasting deeply indented, heavily formatted code into a prompt can use noticeably more tokens than the same logic with consistent, minimal whitespace, and a tokenizer measurement tells you by how much. Numbers are handled differently by different designs: some split long digit strings into fixed-size chunks, some one digit at a time, and some treat common numbers as single pieces. Since the model reasons over those pieces, how a number is cut can affect how easily it can do arithmetic on it, which is one reason that exact calculation is better delegated to code than done in the model's head.

How to measure instead of guess

If you need real counts, compute them with the exact tokenizer and version for the exact model you will call, over representative samples, and record which tokenizer produced them. The implementation section shows how. This chapter does not quote real counts for named tokenizers because a count without the tokenizer name and version is not reproducible.

A model processes a bounded number of tokens at once. Applications divide that bounded space among several competing demands, and the useful way to think about it is as a budget with fixed and variable lines. Take an illustrative model with a limit of 8,192 tokens. A support assistant is configured as follows, all in tokens:

window limit                             8,192
- system instructions                      400
- conversation history                   2,000
- retrieved evidence (policy passages)   3,000
- reserved for the answer                1,000
------------------------------------------------
planned usage = 400 + 2,000 + 3,000 + 1,000 = 6,400
remaining     = 8,192 - 6,400            = 1,792

The 1,792 tokens of headroom is not free to spend freely: it is the room in which history may grow before something must be dropped. The output reservation is the line that is easiest to forget. In many APIs the prompt and the generated answer share the same window, so if the prompt is allowed to fill it, the model has no room left to answer or its answer is cut off. Providers also commonly define separate limits for the window and for the maximum output length, and some counts differ again for any hidden reasoning tokens, so every number here is something to look up for the specific model and provider you use rather than assume. See Context Window for what happens inside the model as the window fills.

Longer contexts are not only a capacity question. Processing time for the prompt grows with its length, memory for cached attention state grows with it, and the practical quality of a model's use of very long inputs is not uniform across the window. The point of budgeting is therefore to include the right tokens, not the maximum number of tokens.

Hosted language-model APIs commonly bill separately for tokens in the prompt (input) and tokens the model generates (output), usually quoted as a price per million tokens, with output priced higher because each output token requires a sequential step. Given a price per million tokens P and a count N, the cost is N / 1,000,000 x P, in the currency of the price. With an illustrative schedule of $2.00 per million input tokens and $8.00 per million output tokens (invented numbers, not any provider's prices):

input  cost = 3,000 / 1,000,000 x $2.00 = $0.006
output cost =   500 / 1,000,000 x $8.00 = $0.004
total       = $0.006 + $0.004          = $0.010

input share of tokens = 3,000 / 3,500 = 85.7%   |  input share of cost = 0.006 / 0.010 = 60%
output share of tokens =  500 / 3,500 = 14.3%   |  output share of cost = 0.004 / 0.010 = 40%

Cost accounting has more structure than the formula suggests, and the details are provider-specific: some providers discount input tokens that were served from a cache of a previously processed identical prefix, some meter hidden reasoning tokens as output even though they are not shown, some bill per request or per image in addition, and some count the framing tokens discussed above. Always compute from the provider's current documented rules and from the usage figures the API returns, not from client-side estimates.

Why conversations get more expensive than they look

A chat model is stateless between calls: to continue a conversation the application sends the history again. Each turn the prompt is therefore the system text plus everything said so far plus the new message. Suppose the system text is 400 tokens, each user message is 150 tokens and each answer is 250 tokens. The input of turn k is 400 + 400(k − 1) + 150, which for turns 1 to 5 is 550, 950, 1,350, 1,750 and 2,150 tokens: a total of 6,750 input tokens billed across the session, while the output totals 5 x 250 = 1,250 tokens. At the illustrative prices that is 6,750 / 1,000,000 x $2.00 = $0.0135 for input plus 1,250 / 1,000,000 x $8.00 = $0.010 for output. The input bill grows with the square of the conversation length, because every turn re-pays for everything before it. Truncating, summarising or caching the shared prefix are the levers, and they are the subject of the production section below and of Caching in AI Systems and Cost Optimization.

Change the number of turns in the figure below: each new turn adds a taller bar, because the whole history is sent again.

Generation is the same vocabulary used in reverse. The model reads the token IDs of the prompt, computes a probability for every entry of the vocabulary as the next token, one token is chosen (greedily or by sampling), its ID is appended to the sequence, and the process repeats until an end-of-sequence token is produced or a length limit is reached. The details of how the distribution is formed and sampled belong to Next-Token Prediction. What matters here is the unit: the model commits to one token at a time, not one word, and the output is the decoded concatenation of the chosen pieces.

Two visible effects follow. Streaming interfaces emit text as tokens arrive, so a long word appears in fragments (refu, nd, then ability) and the user watches words assemble. And because a token can be a fragment of a character's bytes, a client that decodes each token's bytes separately can show garbage for an instant when a multi-byte character straddles two tokens. The fix is to decode a byte stream incrementally:

import codecs

# Hypothetical stream: the 4 bytes of the emoji are split across two chunks.
chunks = [b"Refund sent ", b"\xf0\x9f", b"\x98\x80", b" today"]

print([c.decode("utf-8", errors="replace") for c in chunks])
# ['Refund sent ', '\ufffd', '\ufffd\ufffd', ' today']     <- broken per-chunk decoding

dec = codecs.getincrementaldecoder("utf-8")()
print([dec.decode(c) for c in chunks])
# ['Refund sent ', '', '😀', ' today']                       <- buffered until complete

Generation cost therefore scales with output tokens in both money and latency: each one is a sequential step, so a 500-token answer takes roughly five times as many steps as a 100-token one, whereas the prompt is processed in parallel. This is why constraining an answer's length is one of the cheapest optimisations available, and why a limit on output tokens that is set too low produces a reply that ends mid-sentence with no error.

Budgets are exceeded in practice, not in theory: histories grow, retrieval returns more than expected, users paste large documents. A system must have an explicit policy for what happens when planned usage exceeds the window, and the least careful policy, cutting from the front or the back of the concatenated text, can silently destroy the most important parts of the prompt.

A failure caused by naive truncation

Take the budget above, but let the conversation run longer so that history is 4,100 tokens. Planned usage is now 400 + 4,100 + 3,000 + 1,000 = 8,500 tokens, which is 8,500 − 8,192 = 308 tokens over the limit. A naive implementation concatenates everything in order and drops the first 308 tokens so that it fits. The first 308 tokens belong to the system instructions, which are 400 tokens long. After truncation only the last 400 − 308 = 92 tokens of the instructions survive, and they start in the middle of a sentence.

Suppose the lost part of the instructions held the rules "Only answer from the supplied policy passages" and "Never promise a refund over 100 EUR without approval", while the surviving tail says only "Cite policy IDs." The application throws no error, the model receives a coherent-looking prompt, and the first symptom is a confidently phrased refund promise that violates policy, at a point no log shows. Token-limit failures are dangerous because they are silent. The model was not wrong in any way the platform can see; the prompt was.

Budget-aware packing

The robust approach treats the prompt as a set of components with priorities, not as one string. Fixed, non-negotiable parts (system instructions, the current user message, the output reservation) are reserved first. The remaining room is then allocated to optional parts in order of value: retrieved evidence selected by relevance (see Context Injection and Chunking for how evidence is sized), and the most recent conversation turns. Older turns are dropped as whole turns, never cut mid-message, and may be replaced by a summary, accepting that a summary is a lossy rewrite that can itself introduce errors. If the fixed parts alone do not fit, the request should fail loudly instead of being trimmed.

from dataclasses import dataclass

@dataclass
class Turn:
    role: str
    text: str

def pack_prompt(system, history, evidence, count, window, reserve_output):
    fixed = count(system) + sum(count(e) for e in evidence)
    room = window - reserve_output - fixed
    if room < 0:
        raise ValueError(f"system + evidence exceed the window by {-room} tokens")
    kept, used = [], 0
    for turn in reversed(history):                 # newest first
        cost = count(turn.text)
        if used + cost > room:
            break
        kept.append(turn); used += cost
    kept.reverse()
    return kept, room - used

# Stand-in counter (whitespace words) so the example runs anywhere. It is NOT a real tokenizer.
count = lambda s: len(s.split())
hist = [Turn("user", "a b c d"), Turn("assistant", "e f g"), Turn("user", "h i")]
print(pack_prompt("s s", hist, ["x y z"], count, window=14, reserve_output=3))
# ([Turn(role='assistant', text='e f g'), Turn(role='user', text='h i')], 1)

For the 4,100-token history in the failing example, this policy keeps the system prompt whole (400), keeps the evidence (3,000) and the 1,000-token reservation, and gives history the remaining 8,192 − 400 − 3,000 − 1,000 = 3,792 tokens, which means dropping the oldest turns until at most 3,792 remain. Real systems add a safety margin because counts made on the client can differ slightly from what the provider counts, for instance because of framing tokens. Other levers include retrieving fewer or smaller chunks, moving stable reference material into a cached prefix, and moving long-running state out of the prompt into a store, none of which should be confused with a larger window making the problem disappear.

Tokenization is a representation, not a safeguard. Splitting text into pieces does not neutralise instructions inside it: a document retrieved from the web or a user's message that says "ignore your previous instructions" is turned into tokens just like the system prompt, and the model sees one sequence in which the boundary between trusted and untrusted text is carried only by role markers and by whatever the model learned about them. Prompt injection therefore remains possible whatever the tokenizer is, and defending against it needs separate measures covered in AI Security: least privilege for tools, treating retrieved text as data, output validation and human approval where consequences are real.

Tokenization quirks can also be abused or can cause accidental mismatches. Text that is visually identical can have different bytes (the composed and combining forms of é above, or look-alike characters from different scripts), so a filter that inspects strings for forbidden phrases may not see what the model sees, and a filter working on tokens may be circumvented by changing how a phrase is split. Filtering decisions should be made on normalised text and verified against the same normalisation the model's tokenizer applies.

Finally, a token count is a measure of size, never of quality. Two prompts of 3,000 tokens can differ enormously in the relevance and correctness of what they contain; a larger context filled with marginal passages can degrade an answer while costing more. Fewer, better tokens frequently beat more tokens, and no token statistic says whether the model's answer is true; that needs evaluation against known-correct references.

The practical workflow has four operations: encode text to IDs, inspect the pieces, decode to confirm the round trip, and count against a budget. The example below uses the open-source tiktoken library from OpenAI, whose README documents tiktoken.get_encoding(name), encode and decode and states that encoding is reversible (github.com/openai/tiktoken). It applies to OpenAI-family encodings only: for another model, use the tokenizer published for that model, because counts from one tokenizer are not valid for another. The code below follows the documented calls, but it was not executed while writing this chapter and no counts are claimed.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")       # name of an encoding; pick the one for YOUR model
text = "My order was charged twice; when will the refund arrive?"

ids = enc.encode(text)                           # list[int]
pieces = [enc.decode([i]) for i in ids]          # decode each ID alone to inspect the pieces
assert enc.decode(ids) == text                   # the round trip is lossless

print(len(ids), ids, pieces)

WINDOW, RESERVED_OUTPUT = 8192, 1000             # illustrative limits; read them from provider docs
def fits(prompt_text: str) -> bool:
    return len(enc.encode(prompt_text)) <= WINDOW - RESERVED_OUTPUT

Treat this local count as an estimate for the prompt text, and prefer the usage numbers returned by the provider for billing and for verifying the budget, since chat framing and tool definitions add tokens that the raw text does not contain. Four operational habits are worth adopting: record the tokenizer name with every measurement; keep a test set of representative inputs (your languages, your code, your longest documents) and track their token counts over time; set the output limit deliberately instead of leaving it to a default; and alert when prompts approach the limit, since the truncation failure above is silent.

1. 1

   Measure

   Count real traffic samples with the exact tokenizer of the target model, by component (system, history, evidence, user).
2. 2

   Allocate

   Reserve fixed parts and the output budget first; give each optional component a cap that sums to the window with a safety margin.
3. 3

   Enforce

   Drop or summarise whole units by priority, fail loudly if fixed parts overflow, and log every drop.
4. 4

   Verify

   Reconcile local counts against provider-reported usage; investigate drift after tokenizer or model changes.

Everything in this chapter happens before the neural network runs. After the tokenizer has produced n IDs and the lookup has produced the matrix X of shape [n, d], the model adds information about each token's position (a sequence of identical words must still be told apart by order) and passes the result through its layers, which is the subject of Transformer Basics. Everything the model can do with a prompt, it does with these n rows, so the quality of the vocabulary, the count of rows and the choice of which tokens to include are the first and most controllable levers an application has.

One loose end is worth tying off. The tokenizer and embedding table inside a generative model are not the same as an embedding model used for retrieval; the latter has its own tokenizer and its own vectors, and its output is a single vector for a whole passage. Counts, limits and IDs from one cannot be reused for the other. That pipeline is the topic of RAG Embeddings, and the mechanics of what embeddings are begin in the next lesson.

1. 1

   Token counts are tokenizer-specific. Rules of thumb such as N tokens per word fail on code, under-represented languages and numbers, so count with the exact tokenizer and version of the model you call.

2. 2

   A token is a piece of text, a token ID is its integer address in one tokenizer's vocabulary, and an embedding is the learned vector stored at that address in one model's table; none transfer between models.

3. 3

   Treat the context window as a budget split between instructions, history, retrieved evidence and a reserved output; plan the headroom before history grows past it.

4. 4

   When the prompt does not fit, apply an explicit priority policy: keep system rules, the current message and evidence whole, and trim history. Naive truncation can silently drop instructions such as an approval limit.

5. 5

   Chat templates add special tokens that consume context and are billed as input, and every turn resends the whole history, so cost per turn grows with conversation length.

6. 6

   Log prompt tokens per component, the total and what was dropped, and prefer provider-reported usage for billing; a token count measures size, not quality or safety.