RAG Embeddings Explained

The animation introduces the intuition. This page focuses on embeddings specifically as a RAG ingestion-and-retrieval mechanism rather than on vector-space fundamentals.

RAG uses embeddings so it can find chunks that match the meaning of a question even when they share few words with it. The same embedding model converts every document chunk and the user query into a vector; the chunk vectors are stored in a vector index, and the chunks whose vectors lie closest to the query vector become retrieval candidates. A RAG embedding is a list of numbers, produced by a trained model, so that texts used in similar ways end up with similar lists. Embeddings are a learned tool for ranking likely relevance. They are not a record of facts and their similarity scores are not probabilities that a passage is correct. The general geometry of embeddings is covered in Embeddings Explained; this lesson is about their role inside a RAG system.

A support agent types "customer got double billing". The relevant runbook, RB-PAY-017, says: "Procedure for duplicate payment capture on a single order." The two texts share no content words. A keyword index (the classic inverted index) asks which documents contain "double" or "billing", and returns nothing from the runbook, perhaps surfacing a billing-address FAQ instead. The writer used one vocabulary, the searcher another. People solve this by guessing synonyms; a semantic representation tries to solve it by placing "double billing" and "duplicate payment capture" near each other because they appear in similar contexts across the model's training data.

Keyword search is not obsolete, though. If the agent types the identifier INV-20931, the right document is the one that contains exactly that string, and an embedding model may place many invoice-like identifiers near each other, ranking a neighbor above the match. Semantic and exact matching have opposite strengths, which is why we revisit combining them below and in the Retrieval Pipeline.

The figure below contrasts the two behaviors on the examples above.

An embedding is a fixed-length vector of floating-point numbers, commonly a few hundred to a few thousand dimensions, output by an embedding model. That model was trained on a task that rewards placing related texts close together and unrelated ones far apart, for example by pulling together question–answer pairs or paraphrases and pushing apart mismatched pairs, a process known as contrastive training. Whatever the exact recipe, the model has learned a mapping from text to a point in a high-dimensional space.

Two points are worth holding firmly. First, individual coordinates are not interpretable facts. Dimension 217 is not "about money". Meaning is distributed across the whole vector, and only the geometry (which vectors are near which) carries information. Second, an embedding has no guaranteed relationship to truth. It places "refunds are available within 30 days" and "refunds are not available within 30 days" very close together, because they are about the same topic in the same words. Similar is not the same as equivalent, nor as entailing.

The text is split into tokens by the model's tokenizer. A neural encoder (typically a transformer) reads the tokens together and produces a contextual vector for each one, so that the representation of "capture" depends on whether the sentence is about payments or about photography. Those per-token vectors must be reduced to one vector per text. Many models average them (mean pooling); others use the output at a dedicated summary position. The choice is part of the model and you should not substitute your own. Some models then normalize the result to length 1, which makes dot product equal to cosine similarity; others do not.

Practical consequences follow. Models differ in output dimension (for example 384, 768, or 1,536 — check your model's documentation) and in maximum input length. Text longer than the limit is truncated or rejected, so what you embed is determined by chunking. And because the vector was computed by one specific model, it only makes sense next to vectors from that model.

Real embeddings have hundreds of dimensions that no one designed. To see the geometry, use two made-up dimensions and unit-length vectors. Everything here is illustrative. First, the path through a RAG system, using the runbook chunk and the agent's question from earlier:

indexing:   chunk "Procedure for duplicate payment capture on a single order."
            -> embedding model -> vector p -> stored in the vector index with chunk ID RB-PAY-017#c0003

query:      "customer got double billing"
            -> same embedding model -> vector q -> nearest-neighbor search in the index
            -> RB-PAY-017#c0003 returned as a candidate for the prompt

query                        q = [1.0, 0.0]
relevant policy passage      p = [0.8, 0.6]     |p| = sqrt(0.64 + 0.36) = 1
unrelated passage            u = [0.0, 1.0]

cos(q, p) = (1.0*0.8 + 0.0*0.6) / (1 * 1) = 0.8
cos(q, u) = (1.0*0.0 + 0.0*1.0) / (1 * 1) = 0.0

ranking: p (0.8)  >  u (0.0)

The ranking says p is closer to the query than u. It does not say p answers the question, and 0.8 is not an 80% chance of relevance. In a real model, a score of 0.8 might be excellent for one model and mediocre for another, because each model spreads its vectors differently. Scores are meaningful for ordering candidates under one model, not as absolute truth. The Similarity Search lesson works through cosine, dot product, and distance in detail.

A query like "double billing refund?" is short and interrogative; a document chunk is long and declarative. Some embedding models handle this asymmetry with separate query and document modes, activated by a prefix or instruction (for example, a literal prefix string the model card tells you to prepend), or even with two distinct encoders trained together. If the model expects a prefix and you omit it, or you use the query prefix on documents, quality degrades silently: nothing errors, the scores simply become less meaningful. Always read the model's documentation and encode what it says in one function, so that it cannot differ between ingestion and query.

More fundamentally, vectors from different models live in different spaces. Even two models with the same dimension produce vectors whose coordinates mean different things, so comparing a vector from model A with one from model B yields numbers with no relation to relevance. The same applies to different versions of one model. The rule is to pin the embedding model and version, store them in each record, and refuse to search an index with a query vector from a different model.

Consider two searches against the same knowledge base. For "duplicate payment on one order", embeddings shine: the runbook that says "double capture" is found despite sharing no key phrase. For "order 8472", the useful record is the one that literally contains 8472, and the embedding of the short string "8472" is likely to sit near many other numeric snippets with no special pull toward the right one. A keyword or exact-key lookup is the correct tool.

Production systems therefore use hybrid retrieval: run a lexical search and a vector search, and combine the results. Embeddings extend reach beyond shared vocabulary; lexical and key lookups guarantee precision for identifiers, codes, names, and quotations. Embeddings do not replace deterministic lookup. For something as definite as "get order 8472", the answer is a database query, not similarity.

Three measures are common. Dot product multiplies matching coordinates and sums them; it grows with both alignment and vector length. Cosine similarity divides the dot product by the vectors' lengths, so it measures only direction. Euclidean distance is the straight-line distance between the points; smaller is closer. For unit-length vectors these are tightly related, and rankings agree: maximizing cosine is the same as maximizing dot product and minimizing Euclidean distance. For vectors of varying length they can disagree.

q = [1, 2]      A = [2, 4]      B = [2, 0]      B' = [20, 0]  (B scaled by 10)

dot(q, A)  = 10      cos(q, A)  = 1.000
dot(q, B)  =  2      cos(q, B)  = 0.447
dot(q, B') = 20      cos(q, B') = 0.447      <- same direction as B, still 0.447

by dot product:  B' (20)  >  A (10)  >  B (2)
by cosine:       A (1.0)  >  B = B' (0.447)

Slide the scale of B′ to find where dot product flips the ranking while cosine stays put.

So the metric must match how the model was trained and how the index is configured. If a model produces normalized vectors and recommends dot product, using cosine gives the same ranking. If it produces unnormalized vectors whose length carries meaning, normalizing changes the ranking. Never choose a metric by habit: check the model's documentation, configure the index accordingly, and keep the choice constant between ingestion and query.

One vector per whole document blurs everything into an average and truncates anything past the model's input limit, so a long policy contributes only its opening pages. One vector per chunk is the standard answer, and it means the embedding is only as good as the chunk. If the chunk has the rule without its exception, the vector represents the rule; if it concatenates three topics, the vector sits between them and matches none well. A strong embedding model cannot compensate for a bad split, which is why chunking comes first.

Content type matters too. Tables flattened into numbers and code with unusual identifiers are poorly served by general text embedders; some models are trained for code, and tables often work better when rendered with their headers as sentences. Multilingual corpora need a model that covers each language and, if you want a Spanish query to find an English document, a model trained for cross-lingual alignment. Metadata such as source, date, and permissions should not be baked into the vector but stored alongside it, where filters can use it exactly.

Choose by fit to your task and constraints. Does it handle your domain's vocabulary (legal, medical, code)? Does it support your languages? What are its dimension, its maximum input length, its latency and throughput at your ingestion and query volume, and its cost? Can you host it yourself if privacy rules forbid sending documents to an external API, and does the license allow your use? Higher dimensions generally cost more to store and search (see the arithmetic in Vector Databases).

Public leaderboards (for example the MTEB leaderboard, whose rankings change as models are added) are a useful way to build a shortlist, but they measure performance on public benchmarks. Your documents have internal jargon, product names, and formatting that those benchmarks do not. The deciding evidence should be a labeled set of your own queries and passages, discussed below.

Because vectors from different models are incompatible, switching models means re-embedding every chunk. The migration has to be staged so users never search a half-converted index.

1. 1

   Build B alongside A

   Re-embed every chunk with model B into a new index, recording the model name and version on each record. Keep ingesting document changes into both during the build.
2. 2

   Replay and compare

   Run the labeled queries, and a sample of real recent queries, against both indexes. Compare recall and ranking quality, and check latency and cost.
3. 3

   Switch the query path

   Move the query embedder and the search target to B together, so a model-B query vector is never sent to index A. Invalidate caches that hold old query vectors or results.
4. 4

   Soak and retire

   Keep A for a rollback window. Remove it when B is stable.

Build a labeled set: real or realistic queries, each paired with the passage(s) that should be retrieved. Include synonym queries ("double billing"), exact-identifier queries ("INV-20931"), negative cases where no passage is relevant, and multilingual queries if you support them. Then measure whether the relevant passage appears in the top K results. A tiny worked example with four queries and K = 3:

Q1 "double billing refund"        relevant at rank 1   -> hit,  RR = 1/1 = 1.000
Q2 "invoice INV-20931 status"     not in top 3         -> miss, RR = 0
Q3 "chargeback deadline EU"       relevant at rank 3   -> hit,  RR = 1/3 = 0.333
Q4 "refund to a different card"   relevant at rank 2   -> hit,  RR = 1/2 = 0.500

Recall@3 = hits / queries = 3 / 4 = 0.75
MRR      = (1.000 + 0 + 0.333 + 0.500) / 4 = 0.458

Recall@K tells you whether the retriever's candidates contain what is needed; MRR tells you how near the top it sits. Compare models on identical data and chunks, and look at the individual failures, not just the average. If a model has a good average but misses all identifier queries, you have learned what to cover with lexical search.

The sketch below isolates the details that most often differ between ingestion and query: model pinning, prefixes, batching, and normalization. client.embed is a placeholder for whichever SDK or model server you use; it is not a real library call.

import math

MODEL = "example-embed-v2"             # pinned name and version
QUERY_PREFIX = "query: "               # use exactly what the model card specifies
DOC_PREFIX = "passage: "
BATCH = 64

def l2_normalize(v):
    n = math.sqrt(sum(x * x for x in v))
    if n == 0:
        raise ValueError("zero vector")  # decide explicitly how to handle this
    return [x / n for x in v]

def embed_query(text):
    return l2_normalize(client.embed(model=MODEL, inputs=[QUERY_PREFIX + text])[0])

def embed_documents(texts):
    out = []
    for i in range(0, len(texts), BATCH):
        batch = [DOC_PREFIX + t for t in texts[i:i + BATCH]]   # respect max input length
        out.extend(l2_normalize(v) for v in client.embed(model=MODEL, inputs=batch))
    return out

# store {"embedding": vec, "embedding_model": MODEL} with every record

Embeddings return false neighbors. "How do I cancel a subscription refund?" can land near "How do I cancel my subscription?" because the topics overlap, even though the first concerns money back and the second concerns ending service. Negation, numbers, and conditions are weakly represented: "refunds are allowed" and "refunds are not allowed" embed close together. Domain jargon and new product names that the model has barely seen can map to unreliable positions. Ambiguous questions ("What's the limit?") have no single meaning to encode. Text unlike the model's training data, such as raw logs or unusual formats, may produce vectors that cluster arbitrarily. And when you change the model or its version, behavior drifts in ways that no document edit explains.

The remedies are structural rather than clever. Lexical retrieval anchors exact terms and identifiers. Metadata filters restrict candidates by tenant, version, and date, which the vector cannot express. A reranker reads query and passage together and can separate "cancel subscription" from "cancel and refund". And evaluation on your own labeled data tells you which of these you need.

Once chunks are vectors, you need somewhere to put millions of them and a way to find the nearest ones quickly.