Similarity Search

The animation shows the geometry intuition. This page owns the RAG-specific retrieval role: query embedding, top-K candidate selection, thresholds, and the limits of nearest-neighbor search.

Semantic search in RAG retrieves relevant chunks by meaning rather than by shared words. It embeds the user's query with the same model used for the chunks, scores every candidate (or, with an approximate index, a promising subset) by a similarity or distance function such as cosine, and returns the top-k. For the query "refund for a duplicate payment" the result is a ranked list like this:

query: "refund for a duplicate payment"
1  POL-REF-004#c0007   0.86   duplicate captures are refunded automatically
2  RB-PAY-017#c0003    0.81   procedure for duplicate payment capture
3  FAQ-112#c0001       0.74   general refund overview

The score says how close two vectors are under a particular model and metric. It does not say that the passage is correct, that it answers the question, or that it entails the answer. That gap, between "near" and "useful", is the reason RAG systems add filters, lexical matching, and reranking after the first search. The k-NN and ANN algorithms behind the search are covered in Vectors & Similarity Search.

A support agent asks: "What is the refund process for a duplicate payment?" The search returns a ranked list and the top item is KB-INV-031, "Handling duplicate invoices: how finance merges and voids repeated invoice records". It is genuinely similar to the query: the topic of duplicates in a billing context is shared, and many of the same words appear. It also has nothing to do with returning money to a customer. The actual answer sits in second place, and a stale version of the same policy sits in fifth.

rank  chunk                          score  judgment
1     KB-INV-031#c0002               0.84   not relevant (duplicate invoices)
2     POL-REF-004#v4#c0007           0.82   relevant (current refund policy)
3     FAQ-112#c0001                  0.79   partly relevant (general refund overview)
4     RB-PAY-017#c0003               0.77   relevant (duplicate capture runbook)
5     POL-REF-003#v3#c0007           0.76   stale (superseded policy version)

Closeness is useful: it narrowed millions of chunks to a few dozen worth considering. It is not sufficient: the order within that set was wrong, and the set itself contained a retired version. The sections below explain what the number is, how to compute it, and what to do when it misleads.

1. 1

   Embed the query with the right conventions

   Use the same model and version as the stored chunks, apply the query prefix or instruction the model requires, and normalize if the index expects it. A mismatch yields scores that look normal but rank poorly.
2. 2

   Search

   Compare the query vector with candidate vectors. This is an exact scan for small collections or an approximate index search for large ones.
3. 3

   Return IDs and scores

   The result is an ordered list of chunk IDs with scores. Metadata and text are fetched for the chosen ones.
4. 4

   Hand on

   The list is a set of candidates for the next stages of the retrieval pipeline, not the final evidence.

Keep two things separate. The mathematical similarity between two vectors is exact and deterministic. The search that finds the highest-scoring vectors may be approximate. A surprising result can therefore come from the representation (the model thinks they are close), from the metric, or from the index missing a better match. Diagnosing which one it is requires being able to compute the exact scores for the query yourself.

Cosine similarity measures the angle between two vectors: cos(q, d) = (q · d) / (|q| |d|). The dot product q · d multiplies matching coordinates and adds them; |v| is the vector's length, the square root of the sum of squared coordinates. The result runs from 1 (same direction) through 0 (perpendicular) to −1 (opposite). Use a toy query q = [1, 2] and three documents. The numbers are invented to make the arithmetic visible; real vectors have hundreds of dimensions.

|q| = sqrt(1*1 + 2*2) = sqrt(5) = 2.236

A = [ 2,  4]   q.A = 1*2 + 2*4 = 10    |A| = sqrt(20) = 4.472
               cos = 10 / (2.236 * 4.472) = 10 / 10    = 1.000

B = [ 2,  0]   q.B = 1*2 + 2*0 = 2     |B| = 2
               cos =  2 / (2.236 * 2)  = 2 / 4.472     = 0.447

C = [-1, -2]   q.C = -1 - 4   = -5     |C| = sqrt(5) = 2.236
               cos = -5 / (2.236 * 2.236) = -5 / 5     = -1.000

ranking by cosine:  A (1.000) > B (0.447) > C (-1.000)

If every vector is normalized to length 1, the denominator is 1 and cosine is just the dot product, which is why many pipelines normalize once at ingestion and then use fast dot-product search. Note what cosine does not capture: it knows nothing about how long or detailed a passage is, or whether it is current.

Dot product, q · d, increases with both alignment and length. Euclidean distance, |q − d|, is the straight-line separation between points, where smaller means closer. They can rank the same candidates differently. Take q = [3, 4] with two documents: P = [0.3, 0.4], pointing exactly like the query but short, and R = [5, 5], pointing slightly differently but long.

q = [3, 4]      |q| = 5
P = [0.3, 0.4]  |P| = 0.5            R = [5, 5]  |R| = 7.071

dot:       q.P = 0.9 + 1.6 = 2.5     q.R = 15 + 20 = 35      -> R wins
cosine:    cos(q,P) = 2.5 / (5*0.5) = 1.000
           cos(q,R) = 35 / (5*7.071) = 0.990                  -> P wins
euclid:    |q-P| = |[2.7, 3.6]| = 4.50
           |q-R| = |[-2, -1]|  = 2.236                        -> R wins (smaller)

Figure: the same three vectors in the plane show why the metrics disagree.

Some embedding models are trained so that vector length is incidental, and cosine (or normalized dot product) is the right choice. Others use dot product on unnormalized vectors, where length carries information such as confidence or popularity. Choose the metric the model documents, configure the index to match, and keep that choice fixed between ingestion and search. See Embeddings Explained for how vectors get their lengths.

Exact search scores every vector, keeps the k best, and is guaranteed to return the true top-k for the metric. Approximate search explores a subset chosen by an index structure, trading some recall for speed. The way to know what you are giving up is to run both on a sample of real queries and compute overlap: if the exact top-10 contains 10 items and the ANN top-10 contains 9 of them, recall@10 for that query is 0.9. Average that over many queries and across the filter settings you actually use. The Vector Databases lesson explains how graph and cluster indexes work and why they can miss.

A similarity score is a geometric quantity inside one model's embedding space. It is not a probability that the passage is relevant, and it is not calibrated across models: 0.8 under one model can mean an excellent match and under another can be typical for loosely related text. Scores also shift with the domain, the query length, and the chunking. Practical consequences follow.

You cannot copy a threshold from a blog post or another project. A cutoff such as "discard anything under 0.75" needs to be set on your model and data by looking at labeled relevant and irrelevant results. Even then, an absolute threshold is brittle, since it drifts when you change models or re-chunk. Ranks are usually more stable than scores, which is why strategies that combine several signals often use ranks (as in the reciprocal rank fusion in the Retrieval Pipeline lesson) instead of raw scores.

Top-k decides how many candidates move on. Using the ranked list above, k = 3 sends the irrelevant invoice article, the policy, and the overview FAQ. The runbook at rank 4 is lost, and so is the stale policy at rank 5, which might be a blessing. k = 10 would also bring in the runbook and the stale policy and five more, raising recall but also the downstream cost: if each chunk averages 350 tokens, ten chunks add 3,500 tokens to the prompt versus 1,050 for three, and more irrelevant text for the model to be distracted by. The right k for the first stage is generous (recall matters most), and a later stage cuts to a smaller number.

A search always returns k results if k records exist, however unrelated. A system that treats the best match as the answer will confidently use the least-bad chunk even when the knowledge base says nothing on the topic. Combine a calibrated minimum relevance check (a threshold, or better, a reranker verdict) with filters that include authorization. If nothing authorized and relevant remains, the correct output is an explicit no-result state that leads to abstention or a clarifying question, not a forced answer.

A question is short and unspecific; the passage that answers it is long and full of other details. Embedding models that were trained on question–passage pairs handle this asymmetry, sometimes with distinct query and document modes selected by a prefix or instruction. If you embed a query as if it were a document, or apply a document prefix to a query, results degrade without an error. Instruction-tuned embedders can also accept a task description ("Represent this question for retrieving policy passages") that changes the representation. Keep one code path for query encoding and one for document encoding, and test that each uses the conventions in the model's documentation.

Paraphrases are the strength of semantic search ("double billing" finds "duplicate payment capture"). Several things are its weakness. Exact identifiers, such as a product SKU or error code, are best found by literal matching. Negation is subtle: "refunds are available" and "refunds are not available" are near neighbors. Numbers ("within 5 days" versus "within 50 days") change the answer and barely change the vector. Exceptions and conditions are small phrases inside large passages. Lexical search (such as BM25) rewards literal term overlap and handles identifiers, rare words, and quoted phrases well, while missing paraphrases. Hybrid search combines both ranked lists; the Retrieval Pipeline lesson shows how to fuse them and why raw scores cannot simply be added.

Compressing a passage into a single vector forces it to average its content. A paragraph covering both eligibility and a regional exception yields a vector that represents neither crisply. Multi-vector and late-interaction approaches keep several vectors per passage, typically one per token or per small span, and score a query by matching each of its parts to the best-matching part of the passage and aggregating those matches. This can capture finer detail at the cost of much more storage and more complex search. It is an advanced option to consider when single-vector retrieval is measurably the bottleneck, not a requirement of RAG; often smaller chunks, hybrid search, or a reranker deliver the improvement more cheaply.

Take the ranked list from earlier. In the whole knowledge base, three passages are labeled relevant to this query: POL-REF-004#c0007, RB-PAY-017#c0003, and POL-EU-009#c0002. The search returned five, of which two are relevant.

relevant in corpus (denominator for recall) = 3
retrieved top 5:  KB-INV-031 (N), POL-REF-004 (R), FAQ-112 (N),
                  RB-PAY-017 (R), POL-REF-003 (N)

Precision@3 = relevant in top 3 / 3 = 1 / 3 = 0.333
Recall@3    = relevant in top 3 / total relevant = 1 / 3 = 0.333
Precision@5 = 2 / 5 = 0.400
Recall@5    = 2 / 3 = 0.667
Reciprocal rank = 1 / (rank of first relevant) = 1 / 2 = 0.500

Raising k from 3 to 5 doubled recall (0.333 to 0.667) while precision moved only from 0.333 to 0.400, because the second relevant item happened to sit at rank 4. On another query, the extra results would mostly dilute precision, so look at both numbers and at individual queries. When relevance is graded rather than yes/no, nDCG scores a list by giving higher gains to more relevant items and discounting by rank; the Reranking lesson works one through.

Drag k to see precision, recall and prompt cost move together on the ranked list above.

Return to the list where KB-INV-031 outranked the refund policy. Several remedies address different causes, and you should pick based on evidence from a trace. If the invoice article is simply topically near, a metadata filter on document type or product area (refund and payment policy, not finance-internal procedures) removes it cheaply, if your metadata supports that. Hybrid retrievalwould add a lexical signal where "refund" and "customer" matter. Query reformulation can add context the user left out (for instance rewriting to "refund process for duplicate payment capture on a customer order") but can also change meaning, so measure it. And a reranker that reads the query and each passage together can notice that one passage is about returning money and the other is about merging records, which a single-vector comparison could not distinguish.

Similarity search is one stage. Production retrieval also parses the question, applies permissions, runs several searches, and fuses the results.