Vector Databases

The animation shows the scale problem. This page focuses on the operational question: what vector-search capability you need, and where it can live.

A vector database stores embedding vectors together with their IDs and metadata, and answers the question "which stored vectors are nearest to this query vector, among the records this user may see?" RAG uses one because retrieval has to find semantically related chunks quickly and only among permitted records. The database avoids comparing the query with every vector by building an approximate nearest-neighbor (ANN) index that trades a little recall for large speed gains; the toy graph walk in the HNSW section below shows how that index can miss. Around the index it provides what a RAG system needs in production: upserts and deletes, metadata and ACL filtering, and operational features such as replication and backups. Specifics vary by product, and for a small corpus you may not need a dedicated database at all. Relevance ranking of the returned chunks is covered in Similarity Search, and the index algorithms in Vectors & Similarity Search.

A vector database for an LLM application holds embeddings of your own documents so that, at question time, the system can retrieve the few chunks most related to the question and place them in the prompt. The model then answers from that supplied context instead of relying only on what its weights encode. The database does the retrieval step only: it does not generate text, and ranking and prompt assembly happen around it.

A prototype indexes 1,000 chunks of 768 dimensions. To answer a query it computes a similarity with each stored vector: 1,000 × 768 = 768,000 multiply-adds, which a modern CPU finishes in a fraction of a millisecond. Nobody needs an index for that. Then the knowledge base grows. At 1,000,000 chunks the same query costs 1,000,000 × 768 = 768,000,000 multiply-adds, plus the memory traffic of reading 3 GB of vectors. The cost per query grows linearly with the corpus, so it is slower and more expensive, it competes with other work for memory bandwidth, and it multiplies by every concurrent user. Specialized indexes exist to bend that line.

Two caveats keep this in proportion. Brute-force search is vectorizable and hardware-friendly, and for hundreds of thousands of vectors it can still be the right choice, especially behind restrictive filters that shrink the candidate set. And speed is not the only reason to use a database: lifecycle, filtering, durability, and sharing across services matter just as much.

A vector alone is useless to a RAG system. The stored unit is a record containing the vector, a unique ID, the chunk's text or a reference to where the text lives, the source document and version, filterable metadata, authorization fields, and timestamps:

{
  "id": "POL-REF-004#v4#c0007",
  "vector": [0.0132, -0.0481, "... 768 floats ..."],
  "payload": {
    "source_id": "POL-REF-004", "version": 4, "status": "current",
    "effective_date": "2026-10-01",
    "tenant_id": "acme-eu", "acl": ["support-agent", "support-lead"],
    "heading_path": ["Refunds", "Duplicate charges"],
    "text_ref": "s3://policies/POL-REF-004/v4.json#5120-5894",
    "embedding_model": "example-embed-v2",
    "ingested_at": "2026-10-01T09:04:12Z"
  }
}

It helps to separate three things that are often blurred. A vector index is a data structure that answers nearest-neighbor queries. A vector database wraps one or more such indexes with storage, metadata handling, a query API, durability, and operations. A document store holds full original documents and is usually a different system. Many deployments keep chunk text in the vector database's payload for convenience; others keep only an ID and fetch text from a primary store so that there is one source of truth. Either can be right, but decide deliberately, because it determines where updates and permission changes must be applied.

Exact nearest-neighbor search computes the similarity between the query and every stored vector, then keeps the top k, typically with a heap rather than sorting the whole list. It is simple, deterministic, and returns the true nearest neighbors for the chosen metric. Even when you use approximate search in production, you should keep an exact implementation around: it is the ground truth that lets you measure how much the approximate index loses. The ANN question is never "is it fast?" but "how much recall do I give up for this speed, on my data?"

ANN indexes reorganize vectors so that a query examines only a small, promising fraction of them. They return likely nearest neighbors, so the result can differ from the exact answer, and the fraction of true neighbors recovered is the index's recall. Parameters let you buy recall with more work per query. Two families are common, and products implement different ones with different defaults.

Graph-based indexes such as HNSW (hierarchical navigable small world; Malkov & Yashunin) connect each vector to a handful of near neighbors. A query enters at some node and repeatedly hops to whichever linked neighbor is closer to the query, until no neighbor improves. Layers of coarser graphs let the search take long strides first and refine locally later. Cluster-based indexes such as IVF (inverted file) partition the space into regions around a set of centroids; a query finds the few nearest centroids and scans only the vectors in those regions. Compression techniques such as product quantization (Jégou, Douze & Schmid, IEEE TPAMI 2011) are often combined with either to reduce memory. Reference implementations: FAISS and pgvector.

Do not read general performance claims into any of this. How fast an index is at a given recall depends on the data distribution, dimension, parameters, hardware, and whether filters are applied. Measure on your own data.

The following graph is a simplified illustration, not an implementation of HNSW. Each node is a stored vector. Links can be traversed in either direction (the arrowheads show the direction of the walk below). We hold a query and know each node's distance to it, which a real search would compute as it visits nodes:

1. 1

   Start at A (5.0)

   Neighbors are B (3.2) and C (4.1). B is closer to the query than A, so move to B. C is set aside.
2. 2

   At B (3.2)

   Its neighbor D has distance 1.8, which improves, so move to D.
3. 3

   At D (1.8)

   Neighbors are B (3.2) and E (2.5); neither beats 1.8. The search stops and returns D.
4. 4

   The miss

   G (1.1) was never visited because it hangs off C. A search that keeps a wider candidate list, so that C remains a live option, would reach G. Real indexes expose that knob (often called a search breadth or ef parameter): larger values visit more nodes, raise recall, and cost time.

This is why ANN is a speed–recall trade-off and not an exact algorithm. For RAG, a miss on the true nearest neighbor matters most when the missed chunk was the only one containing the answer, which is why recall should be measured against the exact baseline rather than assumed.

Indexes come into existence by bulk loading (build once from a large set, often the most efficient path) and then change by incremental upserts. Graph indexes support insertion reasonably well; cluster indexes may degrade as new data drifts from the centroids chosen at build time. Deletes are the awkward part: removing a node from a graph can break paths, so many systems mark records as tombstones that are filtered out of results and physically reclaimed later during compaction or rebuild. Until then, deleted data still occupies memory and can influence traversal. Switching embedding models requires an entirely new index, as the Embeddings lesson describes.

t0  v4 records live, status=current
t1  upsert v5 chunks (ids ...#v5#...), status=current       -> both versions searchable
t2  set status=superseded on all v4 chunks                  -> query filter hides v4
t3  background compaction removes superseded records

Stale-result risk: between t1 and t2, or if t2 is lost on a replica, a query can
retrieve v4. Mitigations: filter on effective_date / current-version pointer,
write order that is safe under retries, and a post-retrieval check against the
document registry.

The timeline below shows when each version can be retrieved during the swap.

Freshness has two parts here as well. Your ingestion pipeline determines when new data is submitted; the database determines when it becomes visible to searches. Many systems are eventually consistent or batch index updates, so a record written a second ago may not be searchable yet. Check your product's guarantees rather than assuming read-your-writes behavior.

RAG queries are almost never "nearest in the whole corpus". They are "nearest among chunks this tenant may read, that are current, in this language". How the database combines a filter with an ANN search has large consequences. In post-filtering, the index returns its top candidates and the filter removes those that do not match. If the filter is selective, most candidates may be removed and you can be left with too few results, or none, even though matching records exist. In pre-filtering, the filter limits the search domain first, which is correct but harder to do efficiently inside a graph. Products offer different strategies, such as filtered graph traversal, partitioning by tenant, or switching to exact scan when the filter leaves few vectors. Test how yours behaves with a highly selective filter and check the number of results returned.

Move the slider to see how many of 30 candidates survive each strategy.

{
  "vector": [ ...768 floats from embed_query(...) ],
  "top_k": 30,
  "filter": {
    "must": [
      { "field": "tenant_id", "equals": "acme-eu" },
      { "field": "status", "equals": "current" },
      { "field": "acl", "contains_any": ["support-agent"] }
    ]
  },
  "return": ["id", "score", "payload"]
}

Authorization must not be delegated to the language model. Once an unauthorized chunk is in the prompt, it has been disclosed to a component whose behavior you cannot fully control. Enforce tenant and ACL filters in the database query (and, where required, again against the system of record) so that forbidden chunks are never candidates.

What comes back is a list of IDs with scores and payloads. Keep the trace as well: which filter was applied, how many candidates the index examined, and the scores. When an answer is wrong, the trace tells you whether the right chunk was never a candidate, was filtered out, or ranked too low. Scores depend on the metric and model and are not probabilities; see Similarity Search.

There are several architectural options and no universal winner. A vector extension in a relational database keeps vectors next to your other data, lets you join and transact across them, and avoids a new system to run; the trade-off is that its index options, scale ceiling, and filtering behavior depend on the extension and on how much search load the database should carry. A dedicated vector search engine is built for ANN at scale, with richer index choices and tuning, at the cost of another system to operate and keep consistent with your primary data. A hybrid search platformcombining lexical and vector retrieval in one place simplifies hybrid queries. A local libraryrunning in your process is the simplest for a few hundred thousand vectors or less, but leaves durability, sharing, and updates to you.

Choose by what you must do: whether you need transactions across metadata and vectors, how large the corpus is, how selective your filters are, how fast documents change, what your team can operate, and how much operational risk you will accept. A vector database is a means, not a requirement; plenty of effective RAG systems run on modest indexes.

Vector memory is easy to estimate and often the first surprise. For float32 vectors (4 bytes per number):

1,000,000 vectors x 768 dimensions x 4 bytes
  = 3,072,000,000 bytes  ~= 3.07 GB          (before anything else)

scalar quantization to 1 byte per number:
  1,000,000 x 768 x 1 = 768,000,000 bytes    ~= 0.77 GB   (4x smaller)

Vary the corpus size, quantization and replica count to recompute both the per-query cost and the memory.

Quantization stores each number with fewer bits, and product quantization compresses groups of dimensions into short codes. The trade-off is that distances computed from compressed vectors are approximations, so recall can fall; many systems fetch a short list using compressed vectors and then rescore it with full-precision vectors. Whether the loss is acceptable is an empirical question for your data. Lower-dimension embeddings, when your model offers them, reduce size the same way and may also reduce quality.

Production concerns resemble those of any database. Replication provides availability and read throughput; check whether replicas update synchronously, since asynchronous ones can serve stale or not-yet-deleted data. Sharding or partitioning splits the data across nodes when it does not fit on one machine or when you want tenant isolation; it adds a merge step to query results and complicates rebalancing. Backup and restore should be rehearsed, including how long an index takes to rebuild from source if a snapshot is lost; because the index is derived data, the source documents and the ingestion pipeline are your real backup. Corrupt or partially built indexes need detection and a rebuild path, ideally partial and per-tenant. Watch p50, p95, and p99 query latency, recall on a canary query set, index memory, ingestion lag, and error rates. All of this varies by product, so read the operational documentation of the one you choose.

Benchmark with a representative sample of your vectors and queries, including your real filters. Compute the exact top-k for a set of queries and measure the index's Recall@K against it; a result is only interesting alongside its recall. Record latency at p50, p95 and p99 under realistic concurrency, since tail latency is what users experience. Measure memory, ingestion throughput, and how long a full build takes. Test filter behavior at several selectivities, from "everyone" to "one tenant with 200 records". Test update freshness: how long until a new or deleted record is visible. And estimate operational cost, including replicas and the engineering time to run it. Treat vendor benchmarks as hypotheses to confirm, never as results.

The database can retrieve nearest vectors; the next question is what "nearest" means and why it is not enough.