Embeddings Explained

Watch the journey from failing keyword search to embedding vectors, similarity, and retrieval.

Embeddings in AI are lists of numbers, vectors, that a trained model produces for an input such as a token, a sentence, an image, or an audio clip, arranged so that inputs the model treats as related end up with geometrically related vectors. The numbers are learned parameters or activations, not hand-written features, and individual dimensions rarely have a clean human meaning. What carries information is the relationship between vectors: direction, distance and angle. Those relationships are only meaningful for one model (and often one input mode) at a time, they measure similarity as that model learned it rather than truth, and they have to be compared with the metric the model was trained for. Text embeddings, the vectors for tokens, sentences or documents, are the case this page follows.

• Token → a row of a lookup table, one fixed token embedding per vocabulary entry; the same row wherever the token appears.
• Sentence or passage → one sentence embedding, usually the contextual vectors of its tokens pooled into a single vector; this is what search compares.
• Image → one vector from a vision encoder, which in multimodal models can share a space with text vectors.

Dimensionality is model-dependent: a few hundred to a few thousand numbers per vector is common, and vectors from different models cannot be compared. Inside a transformer, token embeddings are then turned into contextual vectors whose values depend on the surrounding words.

This page builds the idea from the bottom: a lookup table that turns token IDs into vectors, how context changes those vectors, how training shapes the geometry, how to compute and compare similarity by hand, and what it takes to generate, evaluate, store and migrate embeddings in a real system. How the nearest vectors are found quickly at scale is the subject of Vectors & Similarity Search, and how embeddings are used for indexing and retrieval in a RAG system is covered in RAG Embeddings.

A customer types money taken twice into the help-centre search box. The policy that answers them is titled Duplicate payment reversal. A keyword engine tokenises both strings and compares the terms:

query terms:   {money, taken, twice}
title terms:   {duplicate, payment, reversal}
intersection:  {}            -> keyword score 0

The two strings share no word, so a pure lexical match scores zero even though a person would pair them immediately: “taken twice” and “duplicate” describe the same event, “money” and “payment” sit in the same domain, and “reversal” is the remedy the customer wants. Engineers patch this with synonym lists, which grow without end and still miss phrasings nobody anticipated.

A learned representation attacks the problem differently. Instead of comparing spellings, a model maps each text to a point in a space where the model has learned, from a very large number of examples, that texts used in similar ways should sit near each other. Then “are these related?” becomes “are these points close?”, a numeric question. The rest of the page makes that sentence precise and also marks where it stops being true: the same machinery will happily rate a policy for the wrong country as very similar to the right one, and nothing in a similarity score tells you which is correct.

A vector is an ordered list of d numbers, here a point (or arrow from the origin) in d-dimensional space. A vector embedding is a vector that a model has learned to produce so that its geometry is useful for some purpose. The word covers several different artefacts that are not interchangeable:

• Token embeddings are rows of a table inside a language model, one per vocabulary entry. They are the first layer's input; they describe a token in isolation (next section).
• Contextual (hidden-state) embeddings are the vectors a transformer holds for each token after one or more layers of mixing with the surrounding tokens.
• Sentence or document embeddings compress a whole span of text into a single vector, typically by pooling contextual vectors, and are trained so that the result is useful for comparing whole texts. This is what a search system stores.
• Image, audio and multimodal embeddings come from encoders for those inputs. A multimodal model is trained so that a photo and a caption that describe each other land near each other, which is what lets text search images; see Multimodal AI.

Mixing these up causes real bugs: the first token-embedding row for “refund” is not a usable document-search vector, and a vector from a text-only model cannot be compared with one from a multimodal model unless that model was built to share a space. When this page says “embedding” without qualification it means a sentence/document-level vector from a dedicated embedding model.

Language models cannot do arithmetic on text; they start from integers. A tokenizer (see Tokens Explained) turns money taken twice into token IDs. Token IDs are just names; ID 4411 is not “bigger” than ID 17 in any meaningful way. The model's first step converts each ID into a vector by indexing a learned matrix:

E ∈ R^V×d,   x_t = E[id_t] ∈ R^d

V is the vocabulary size (number of distinct token IDs), d is the embedding dimension,id_t is the ID at position t, and E[i] is row i. With the toy sizes V = 10,000 and d = 128 the table has 10,000 × 128 = 1,280,000 scalar parameters; at 4 bytes each (float32) that is 5,120,000 bytes, about 5.1 MB. Real vocabularies and dimensions differ by model and are much larger in many cases, but the shape logic is identical.

The figure shows the whole operation. There is no multiplication and no knowledge of neighbours: row 17 is copied out for “money” whether the sentence is about a bank account or a movie budget. (Implementations often write this as a one-hot vector times E, which is mathematically the same and computationally just a row read.) The values in E are not designed. They start random and are adjusted by gradient descent together with the rest of the network, as described in Training vs Inference, because rows that help predict text, or help match queries to documents, are rewarded. In production use the table is frozen and only read.

Word2Vec (Mikolov et al., arXiv:1301.3781) popularised exactly this idea for words: train a small model on a prediction task over a large corpus, and keep the learned table as a by-product. Its famous vector-arithmetic examples were observations about a trained space, not guarantees of how embeddings behave in general.

Consider bank in “the river bank” and in “the bank account”. The lookup gives the same row for both, so a static embedding has to be a compromise between meanings: in a toy 2-D space, a point that is moderately near both finance and river contexts. A transformer fixes this by treating the lookup as only the starting point. Each layer lets every position mix in information from the other positions (the mechanism is attention; the overall stack is in Transformer Basics), so the vector for bank after a few layers depends on its neighbours.

To see the effect without the machinery, use a deliberately crude stand-in that is not how a transformer computes: replace each token's vector by half itself plus half its neighbour's. Take the toy static vectors bank = (0.6, 0.6), river = (0.9, 0.1) and account = (0.1, 0.9).

bank in "river bank":    0.5*(0.6, 0.6) + 0.5*(0.9, 0.1) = (0.75, 0.35)
bank in "bank account":  0.5*(0.6, 0.6) + 0.5*(0.1, 0.9) = (0.35, 0.75)

cosine(static bank,  river)         = 0.781   cosine(static bank, account) = 0.781
cosine(bank | river, river)         = 0.947
cosine(bank | account, account)     = 0.947
cosine(bank | river, bank | account) = 0.766

The static row is equally close to both context words (0.781 each); after mixing, each contextualbank is clearly closer to its own context (0.947) and the two senses have moved apart (0.766). A real transformer learns how to mix, with different weights per layer and head, rather than using a fixed 50/50 average, but the observable consequence is this one: the same token gets different vectors in different sentences.

Pooling: one vector for a whole text

A search index wants one vector per chunk, not one per token. Embedding models produce it by pooling the contextual token vectors, for example by averaging them or by reading the vector at a designated summary position, often followed by normalisation to unit length. Mean pooling of three toy token vectors (1,0), (0,1), (1,1) gives ((1+0+1)/3, (0+1+1)/3) = (0.667, 0.667). Which pooling a model uses is part of how it was trained, so you use the provider's pipeline as documented rather than pooling hidden states yourself. Two consequences follow: very long inputs are summarised into the same fixed number of dimensions as short ones, so detail is lost; and inputs beyond the model's maximum length are truncated or rejected, so anything after the cut is simply not represented.

Nobody tells the model that “money taken twice” belongs next to “duplicate payment reversal”. A retrieval embedding model is typically trained with a contrastive objective over a very large set of text pairs that are known or assumed to belong together (a question and the passage that answers it, a title and its body, two paraphrases). For each query q with a matching document d⁺, the training compares it against non-matching documents d⁻, often the other examples in the same batch, and updates the model so that the matching pair scores higher than the others. A common form of the loss for one query is:

L = −log( exp(s(q,d⁺)/τ) / ( exp(s(q,d⁺)/τ) + Σ_j exp(s(q,d⁻_j)/τ) ) )

s is a similarity (often cosine) and τ is a temperature that controls how sharply the loss separates scores. With s⁺ = 0.8 and two negatives scoring 0.3 and 0.1, and τ = 0.1, the numerators are e⁸ ≈ 2981, e³ ≈ 20.1 and e¹ ≈ 2.72, so the matching document gets 2981/3004 ≈ 0.992 of the mass and the loss is −log(0.992) ≈ 0.0076: the model is already doing well on this example. With a larger temperature τ = 1 the same scores give a share of 0.475 and a loss of 0.743, a much weaker signal. If the negatives had scored 0.75, the loss would be large and gradients would pull the query and the wrong document apart. Training on such many examples pushes matched pairs together and unrelated ones apart in the same space, which is why the geometry becomes useful for retrieval.

Two details matter for engineers. First, hard negatives, wrong documents that look plausible (the policy for another country, a superseded version), are what teach a model to distinguish near-misses; a model trained mostly on easy negatives may separate topics well and still confuse close variants. Second, the dimensions themselves are not interpretable axes. The training objective constrains distances and angles, not coordinates; rotate the whole space and every score is unchanged. So there is no “money dimension” to read off. Labelled axes in diagrams, including the toy ones on this page, are drawing conveniences.

The most common comparison for text embeddings is cosine similarity, the cosine of the angle between two vectors:

cos(a, b) = (a · b) / (‖a‖ ‖b‖),   a · b = Σ_i a_i b_i,   ‖a‖ = √(Σ_i a_i²)

Take three 2-D vectors: a = (1, 2), b = (2, 4), c = (2, 0).

1. 1

   a versus b

   a · b = 1·2 + 2·4 = 10. ‖a‖ = √5 ≈ 2.236, ‖b‖ = √20 ≈ 4.472, product = √100 = 10. So cos(a, b) = 10/10 = 1.0: the angle is 0°. b is just a scaled by 2.
2. 2

   a versus c

   a · c = 1·2 + 2·0 = 2. ‖c‖ = 2, so the denominator is 2·√5 ≈ 4.472 and cos(a, c) = 2/4.472 ≈ 0.447. The angle is arccos(0.447) ≈ 63.4°.
3. 3

   Read the result

   Under cosine, b is a perfect match for a and c is a weak one. The score depends on direction only; doubling a vector's length changes nothing.

Scale invariance is useful for text: a long document and a short one on the same subject can produce vectors of different lengths but similar directions. It is also a limit. Cosine ignores magnitude entirely, so any information a model stores in vector length is discarded, and the score is relative to the model: 0.78 from one model may be a strong match while 0.78 from another is lukewarm. A cosine score is not a probability, not a percentage of “meaning shared”, and not a measure of whether a passage is true or answers the question. In high-dimensional text spaces scores for genuinely related texts commonly cluster in a narrow range well above zero, so thresholds have to be calibrated per model on labelled data.

Three measures are in everyday use: cosine similarity (angle), Euclidean distance ‖a − b‖ (straight-line gap; smaller is closer) and the plain dot product a · b (larger is closer). They coincide when vectors are normalised to length 1: then ‖a−b‖² = 2 − 2·cos(a,b) and a·b = cos(a,b), so all three give the same ranking. They diverge when lengths differ, which the same toy vectors show: ‖a − b‖ = √(1+4) ≈ 2.236 and ‖a − c‖ = √(1+4) ≈ 2.236. Euclidean distance calls b and c equally far from a, while cosine says b is identical in direction and c is not.

A counterexample where magnitude hijacks dot-product ranking: a query q = (1, 1) and two documents d1 = (1, 1.2) (nearly the same direction, length 1.56) and d2 = (10, 0) (a different direction, length 10).

            dot(q,d)   cosine(q,d)   euclid(q,d)
d1 (1,1.2)    2.2        0.996         0.20        <- most similar by direction
d2 (10,0)    10.0        0.707         9.06

dot product ranks d2 first (10 > 2.2); cosine and Euclid rank d1 first.

In the figure below, slide the length of d2 while its direction stays fixed: cosine and Euclidean never change their verdict, but the dot product flips as soon as d2 is longer than about 2.2.

If an index uses raw dot product and the model produced vectors whose length varies with, say, how common a passage's vocabulary is, high-norm vectors will dominate results regardless of topic. Some models are in fact trained for dot product and give length a meaning, so the rule is not “always use cosine” but use the metric the model was trained and documented for, and configure the index to match. When a provider returns unit-length vectors, cosine, dot product and Euclidean are interchangeable and dot product is the cheapest, which is why many systems normalise at write time. Detailed metric comparison and index behaviour continue in Vectors & Similarity Search.

Every plot on this page uses two dimensions because paper has two. Real embedding vectors have hundreds to a few thousand dimensions, and high-dimensional geometry does not behave like a plane. A sharp illustration: two independent random 768-dimensional Gaussian vectors have a cosine that averages 0 with a standard deviation of about 1/√768 ≈ 0.036 (a quick simulation of 2,000 pairs gives 0.036). Unrelated points are almost all nearly perpendicular, and there is room for an enormous number of such directions, which is partly why one space can hold so many distinct topics. It also means that when real text pairs score, say, 0.3 or 0.6, those are very large departures from chance in this space, but only the model-specific calibration can tell you what they mean.

When people want to see an embedding space they project it to 2-D with PCA (a linear projection onto the directions of greatest variance), t-SNE or UMAP (non-linear methods that try to preserve neighbourhoods). Every such projection throws information away, and the picture can mislead: clusters may appear that are not well separated in the full space, truly distant points may land together, and for t-SNE and UMAP the distances between clusters and the apparent sizes of clusters are not reliable. The smallest possible demonstration is three points in 3-D.

Use projections to explore (spot an obviously odd batch, check that a language or product line forms its own region), never as evidence that retrieval works. Evidence is measured on labelled queries, below.

An embedding is only useful inside a pipeline that decides what gets embedded, keeps track of where each vector came from, and embeds queries the same way as documents. The offline half runs when content changes; the online half runs per request.

Three points in the diagram are easy to miss. Chunking exists because a model has a maximum input length and because one vector for a 40-page document blurs every topic in it; where to split is a retrieval design question covered in Chunking. Metadata and provenance (document ID, section, version, owner, permissions, which embedding model and version produced the vector) are what let you filter, cite and later migrate; a vector without them is a number you cannot trace. And the query is embedded by the same model, in the correct mode. Many retrieval models distinguish query text from document text, using different prefixes or API parameters, because a short question and a long answer passage are different kinds of text; skipping the documented mode quietly degrades quality. The domain-specific end of this pipeline, including vector-database behaviour, is in RAG Embeddings and Vector Databases.

Selecting an embedding model is an engineering trade-off between several properties that have to be checked against your data rather than assumed from a leaderboard: whether it handles your languages and domain vocabulary (legal clauses and error codes are far from news text); its dimension, which sets storage and search cost; its maximum input length and truncation behaviour; whether it expects separate query and document modes; cost and throughput for the volume you embed at ingestion and per query; and its version policy, since a provider may retire or update a model.

The rule that dominates operations is that embeddings from different models are not comparable. Each training run finds its own arbitrary coordinate system. A model could have learned a perfectly good geometry and then be rotated, and nothing about the “meaning” would change, yet the coordinates would. Two models with different dimensions cannot even be compared numerically (a 768-vector against a 1024-vector is a shape error), and two models with the same dimension will produce meaningless scores, not an error, which is the dangerous case. Even a new version of the same model family can shift the space. Therefore each stored vector must carry the identity of the model and version that produced it, and queries must use that same one.

“It looks right on a few examples” is not evaluation. A retrieval evaluation needs a set of real or realistic queries, a labelled set of documents judged relevant for each, and metrics computed on the ranked list the system returns. Three are standard. Recall@k is the fraction of a query's relevant documents found in the top k. Reciprocal rank (averaged over queries to give MRR) is 1/rank of the first relevant result. nDCG@k rewards putting relevant items high, discounting each by log₂(rank+1) and dividing by the best achievable score.

Suppose for the query “money taken twice” the labels say three documents are relevant (D2, D4, D9), and the system returns this top 5:

rank  doc  relevant?
1     D7   no   (refund timing FAQ)
2     D2   YES  (duplicate payment reversal)
3     D5   no   (card fraud)
4     D4   YES  (duplicate-charge troubleshooting)
5     D1   no   (shipping delays)
(D9 is relevant but was not retrieved in the top 5)

recall@3 = 1 relevant in top 3 / 3 relevant total = 0.333
recall@5 = 2/3 = 0.667
reciprocal rank = 1/2 = 0.5
DCG@5   = 1/log2(3) + 1/log2(5) = 0.631 + 0.431 = 1.062
IDCG@5  = 1/log2(2) + 1/log2(3) + 1/log2(4) = 1 + 0.631 + 0.5 = 2.131
nDCG@5  = 1.062 / 2.131 = 0.498

With four queries whose reciprocal ranks are 0.5, 1, 0.333 and 0 (no relevant result retrieved), MRR is 1.833/4 ≈ 0.458. A single aggregate hides structure, so inspect failures: the zero is the interesting query. Build the set so that it includes hard negatives (the wrong-country policy, the superseded version), identifiers and dates, rare vocabulary, and different user groups and languages, because an average over easy queries says little about the cases that cause incidents. Re-run it whenever the model, chunking or index parameters change, and again after your content drifts, since a model that was good on last year's documents can degrade on new product names (domain shift). Embedding models also inherit biases of their training data, so check that retrieval quality does not differ systematically across languages, dialects or user groups.

import math

def recall_at_k(ranked, relevant, k):
    return len(set(ranked[:k]) & set(relevant)) / len(relevant)

def reciprocal_rank(ranked, relevant):
    for i, doc in enumerate(ranked, start=1):
        if doc in relevant:
            return 1 / i
    return 0.0

def ndcg_at_k(ranked, relevant, k):
    dcg = sum(1 / math.log2(i + 1) for i, d in enumerate(ranked[:k], 1) if d in relevant)
    ideal = sum(1 / math.log2(i + 1) for i in range(1, min(len(relevant), k) + 1))
    return dcg / ideal

ranked = ["D7", "D2", "D5", "D4", "D1"]
relevant = {"D2", "D4", "D9"}
print(round(recall_at_k(ranked, relevant, 3), 3))   # 0.333
print(round(recall_at_k(ranked, relevant, 5), 3))   # 0.667
print(round(reciprocal_rank(ranked, relevant), 3))  # 0.5
print(round(ndcg_at_k(ranked, relevant, 5), 3))     # 0.498

Vectors are bulky. For N vectors of dimension d stored as float32 (4 bytes per number), raw size is N × d × 4 bytes. For N = 1,000,000 and d = 768: 1,000,000 × 768 × 4 = 3,072,000,000 bytes ≈ 3.072 GB (decimal gigabytes; ≈ 2.86 GiB in binary units). That is raw vectors only; the search index structure, metadata, text, and every replica add to it. A 1024-dimensional model at the same scale would need 4.096 GB, and storing the same 768 values as 1-byte integers would need 0.768 GB, at some cost in precision that must again be measured.

Drag the dimension in the figure below to see how raw size scales for a million vectors at three precisions.

Because vectors from different models cannot be mixed, changing models means re-embedding the entire corpus. Doing that in place is dangerous: while it runs, an index holds a blend of old and new vectors and search quality becomes unpredictable, and if the new model is worse there is nothing to go back to. The safe pattern is a dual-index migration.

Steps: write v2 vectors into a new index whose schema declares its own dimension (an index created for 768 will reject or mis-handle 1024-length vectors, which is a dimension mismatch you want to hit in staging); backfill in batches, tracking which documents are done, and apply live document changes to both indexes while the backfill runs; run the labelled evaluation against both; switch serving with a flag so the change is reversible in seconds; and keep v1 for an agreed retention period before deleting it. Mid-migration you pay for two indexes, so the capacity plan should include 3.072 GB (v1 raw) + 4.096 GB (v2 raw) = 7.168 GB before index overhead and replicas. Rollback is trivial only because the old index still exists and the old query model is still callable.

Beyond search, the same vectors support clustering (grouping support tickets by theme without predefined labels), near-duplicate detection (flagging articles with cosine above a threshold tuned on labelled duplicates), recommendations (items near those a user engaged with) and classification features (feeding vectors to a small classifier, which can need far fewer labels than training from raw text). All of them rest on the same contract: nearby means similar as the model learned similarity.

That contract breaks for questions where the answer is exact. An order number, an error code like E-4021, a date or a version string may be tokenised into pieces whose vector differs little from a neighbouring identifier, so “order 48217” and “order 48271” can look nearly identical to an embedding while being different orders. Negation and numbers are similarly weak: “refund approved” and “refund not approved” often sit very close, because they appear in the same contexts. And permissions are not semantic: whether a user may see a document is a rule, not a similarity. For such needs, combine embeddings with exact filters, keyword or identifier lookup, and metadata predicates, a hybrid design where similarity proposes candidates and deterministic rules decide what is allowed and exact. Reordering candidates with a stronger model is covered in Reranking.

A vector looks like harmless noise, but it is derived from the text and carries a great deal of it. It is not anonymised: research has shown (for example Morris et al., arXiv:2310.06816) that, given access to the vectors and the embedding model, an attacker can recover approximate text or infer attributes of the source, so a leaked vector store should be treated like leaked documents. Design on that basis.

• Enforce access control at query time. Store each chunk's ACL or tenant ID as metadata and apply it as a filter in the same search, before results leave the index. Similarity must never decide who may see a chunk, and post-filtering a top-k list after the fact can return empty or wrong pages.
• Isolate tenants. Separate indexes or enforced namespaces per tenant, so a missing filter in one code path does not expose another customer's data.
• Honour retention and deletion. When a document or a user's data must be deleted, delete its vectors, its chunks and any caches, including old model versions kept for rollback, and record this. Re-embedding does not remove the old vectors unless you delete them.
• Minimise what you send. Text sent to a hosted embedding service leaves your boundary; check the provider's data-use and retention terms and redact where needed.

The broader view of securing AI systems is in AI Security.

With a trusted embedding model, a labelled evaluation set and compatible vectors in an index, the remaining question is purely computational: given a query vector, find the nearest stored vectors fast enough to serve a request. Comparing against all N vectors costs N × d multiply-adds per query; for 1,000,000 vectors of 768 dimensions that is 768 million multiply-adds, which is why production systems use approximate indexes that trade a small, measurable loss of recall for large speed-ups. That trade-off, along with the metrics and the index families, is the next lesson. If your goal is grounding a language model's answers in your documents, the retrieval-specific view of these ideas continues in RAG Similarity Search.

1. 1

   Vectors from different models are never comparable, so changing the model means re-embedding the whole corpus into a new index and cutting over, not mixing versions in place.

2. 2

   Use the metric the model was trained for. A raw dot product on vectors whose length varies can rank a high-norm document above a better-aligned one, while cosine and Euclidean distance do not change their verdict with length.

3. 3

   Judge a model on your own queries with labelled relevant documents (recall@k, MRR); a few good-looking examples are not evaluation, and retrieval metrics do not show whether a passage is correct, current or permitted.

4. 4

   Similarity is the wrong tool for exact tokens such as order numbers, error codes, dates and version strings; pair it with exact or keyword matching.

5. 5

   Raw storage is N × d × 4 bytes in float32, so 1,000,000 vectors of 768 dimensions is about 3 GB before index overhead; plan precision and dimension up front.

6. 6

   Treat vectors as data derived from the source text: apply access-control filters inside the search, isolate tenants, delete vectors and caches along with documents, and project to 2-D only to explore, never as evidence of retrieval quality.