Why RAG Exists

Watch how a normal workplace question exposes the gap between a model's training data and the information a real system needs at answer time.

Retrieval-augmented generation (RAG) exists because a language model cannot be trusted to recall facts that change, that are private, or that must be traceable to a source. A RAG system looks up relevant evidence at question time, places that evidence in the model's input, and asks the model to answer from it. The model still does the language work; the lookup supplies the facts.

So why use RAG instead of fine-tuning an LLM? Use RAG when the answer depends on private or frequently updated factual knowledge that must be cited, because the documents can change without retraining anything. Use fine-tuning when you need to change behavior, such as tone, output format, or a narrow skill. RAG is the right tool when the answer lives in a body of documents that changes, is too large to paste into every prompt, and has to be attributable. It is the wrong one for live account data or for a short, stable set of rules.

A payments company runs an internal support assistant. A customer writes: "I was charged twice for order 8472. Am I eligible for a refund, and what should support do?" The assistant is a general-purpose model with a polite system prompt and nothing else. It answers:

Yes. Duplicate charges are refundable. Ask the customer to submit a refund
request form, and refunds are issued within 10 business days.

The refund policy was revised on 2026-10-01. Version 4 (POL-REF-004) says confirmed duplicate captures are refunded automatically within 5 business days, with no request form, and adds an exception for customers covered by EU consumer rules. Version 3 (POL-REF-003), with the form and the 10 days, is what the model's training data happened to contain, or what it reconstructed as the typical shape of a refund policy. An agent who follows the answer tells the customer to fill in a form that no longer exists and quotes a deadline that is twice as long as the commitment the company now makes.

Nothing was malfunctioning. The model produced the most plausible continuation of a refund question, and plausible text carries no information about whether it is current. Eloquence is a property of the writing; evidence is a property of the source. A system that needs the second cannot be built by improving the first.

When a model answers, it can draw on exactly three things. The first is its parameters: billions of weights adjusted during training. These encode statistical regularities of the training text, and some facts can be recovered from them, but the weights are not a database. They do not store a verbatim copy of every source, they cannot be queried for provenance, and a fact that appeared rarely or inconsistently may come back blurred or wrong. The second is the current input: the system prompt, the conversation, and anything pasted in, bounded by the context window. The third is any tool results that the application deliberately returns into that input, such as the output of an API call.

Your enterprise knowledge base is none of these by default. Training finished before your latest policy existed, and your private documents were never in the training set. The knowledge cutoff gives you stale knowledge, not merely missing knowledge: an older version of your policy, or a public document that looks similar, can be absorbed and reproduced with the same confidence as a current one. There is also no source attached. When the model says "10 business days", you cannot ask which document or which version that came from, because it did not look anything up. This lack of attribution is a core reason these answers are hard to audit, and it is the root of the failures described in the Hallucinations lesson.

The figure below shows the three channels and where a RAG system and a tool attach.

Everything RAG does follows from these three channels. It does not change the weights. It uses the second channel, the input, to deliver the right facts at the right moment.

The obvious alternative is to skip retrieval and include every policy document in every prompt. It is worth taking seriously, because for small corpora it is the correct answer, and understanding where it breaks tells you what retrieval is for.

Suppose the support knowledge base has 600 documents averaging 3,000 tokens: 1.8 million tokens. That does not fit in most context windows today, so you would have to choose what to include, and choosing is retrieval. Even if a corpus does fit, you pay for every input token on every request, and processing a long prompt adds latency before the first output token appears. A question that needs two paragraphs of evidence would be billed and delayed as if it needed the whole library. Models also do not use all positions in a long input equally well; relevant text surrounded by thousands of tokens of unrelated policy can be overlooked.

Slide the corpus size to see what each request would carry.

Version conflicts are a quieter problem. A knowledge base usually contains several revisions of the same policy, drafts, regional addenda, and superseded pages. If everything is pasted in, the model must decide which statement wins, and nothing in the raw text tells it that POL-REF-003 is retired. Finally, there is authorization. A prompt that contains every document gives every user whatever the model chooses to reveal. Access control must be enforced before text reaches the model, which means somebody has to select, per user, which documents are even eligible. That selection step is again retrieval.

So the honest boundary is this: when the whole relevant corpus is small, stable, and visible to every user, put it in the prompt (ideally with prompt caching if your provider supports it) and stop. When it is large, changing, or permissioned, you need a step that chooses what goes in.

RAG separates two jobs the model is bad at combining: knowing and writing. Knowledge moves into a store you control, where it can be updated, versioned, permissioned, and cited. At query time the system retrieves the few passages relevant to this question, supplies them as evidence, and instructs the model to answer from that evidence and to say so when it is insufficient.

Notice what this definition does not say. It does not say the store is a vector database, and it does not say embeddings are involved. The evidence might come from keyword search over an inverted index, a SQL query, a vector similarity search, or a combination. In practice many systems use dense vector search together with lexical search because each catches things the other misses. The RAG Pipeline lesson draws the whole system, and later lessons explain each part. For now, "retrieval" means any mechanism that chooses evidence for this question.

Here is the same support request with retrieval in front of the model. The system searches the policy collection and finds one passage that matches the question about duplicate charges. It arrives with metadata that the application, not the model, attached at indexing time:

[E1] source_id: POL-REF-004   version: 4   effective: 2026-10-01
     heading: "Refunds > Duplicate charges"
     text: "When two settled captures exist for the same order reference,
            the second capture is refunded automatically within 5 business
            days. No refund request form is required. Customers covered by
            EU consumer rules may additionally request a written
            confirmation of the refund."

The assembled prompt tells the model to answer only from the numbered evidence and to cite it. The response becomes:

Under the current refund policy, a confirmed duplicate capture on the same
order is refunded automatically within 5 business days, and no request form
is needed [E1]. Support should therefore not ask the customer to fill in a
form.

I cannot tell from the available documents whether order 8472 was actually
charged twice. That needs a lookup in the payments system. If two settled
captures are confirmed, the policy above applies.

Three things changed. The deadline and the missing form match version 4, because that is the text the model was shown. The claim carries a marker, so a reviewer can open POL-REF-004 and check. And the answer separates what the documents establish from what they cannot: nothing retrieved says anything about order 8472, so the model was told to state that gap rather than infer it. That last behavior does not appear by itself; the instruction to abstain, and a test that confirms it holds, are part of the system. The Context Injection lesson covers how to build it.

Four problems motivate RAG. Each is improved by retrieval, and each leaves a residual failure that you have to engineer for.

Freshness

If the index contains version 4, the answer can reflect version 4 without retraining anything. But a model is not automatically current: the index is only as fresh as the ingestion job that last ran. If the policy is edited at 09:00 and the nightly sync runs at 02:00, the answer is wrong for 17 hours even though the system is "RAG-based". Freshness in RAG is measured as lag between a source change and the moment queries can see it, and it needs a number and an alert.

Private and domain-specific knowledge

Internal policies, tickets, and runbooks can be searched without ever being used for training. The residual risk is access: a retriever that finds a document the user may not read has leaked it into the prompt, and possibly into the answer. Permissions have to be enforced in the retrieval layer against a trusted identity, not requested politely in the prompt.

Traceability

Because evidence has an identifier, an answer can point to it. The residual risk is that a citation marker proves only that the model wrote a marker. It does not prove that the cited passage supports the sentence. A careful system checks claims against the cited text rather than trusting the marker.

Selective context use

Sending two or three relevant passages instead of a whole library reduces cost and distraction. The residual risk is that the right passage may not be selected at all. If retrieval misses the one clause that contains the exception, the model sees a plausible but incomplete picture and answers confidently from it. Retrieval relevance is also not truth: a retrieved passage can be outdated, wrong, or contradicted elsewhere.

These approaches are often presented as competitors, but they answer different questions. A useful way to choose is to ask, for the information the answer depends on: how big is it, how often does it change, does it need to be exact, and who may see it?

A direct call is fine for general explanations, drafting, and tasks where being slightly off is cheap. Long-context prompting suits a bounded, stable, non-sensitive set such as one product manual. Deterministic tools are required whenever the answer is an exact value held by a system of record: a balance, an order status, whether two captures exist. A model asked to guess these will guess, and a document store cannot tell it. Fine-tuning, covered in the Fine-Tuning lesson, shapes behavior such as tone, format, or a narrow skill. It is a poor way to carry facts that change monthly, because every update becomes a training run and the result still has no citations. RAG is for locating and grounding answers in documents that evolve.

These combine well. In our running case, the policy passage about duplicate captures belongs in retrieval, and the question of whether order 8472 has two settled captures belongs in an authorized payments lookup that returns structured data. The best answer uses both: the tool result establishes the fact, the retrieved policy establishes the consequence, and the model composes them. Neither alone can answer the original request.

Figure: the running case as one request that combines a tool call and retrieval.

RAG replaces one problem with an engineering system, and it is better to anticipate its parts as consequences than to meet them as surprises. Documents must be brought in from where they live, so you need connectors and an ingestion job. Files must be parsed, and a badly parsed PDF puts garbled text into your index. Text must be split into retrievable units, and where you cut changes what can be found (see Chunking Strategies). If you use semantic search, text must be converted to vectors by an embedding model that you must version and eventually migrate (see Embeddings Explained). Every unit needs metadata that records its source, version, and who may read it, because updates, deletes, and permission checks all depend on it.

Then there is the quality of retrieval itself, which you must measure rather than assume, and the observability to tell, when an answer is wrong, whether retrieval missed, the evidence was stale, the prompt dropped it, or the model ignored it. Each of those is a different bug with a different fix. There is also ongoing cost: indexing compute, storage, per-query search, and the extra input tokens on every generation. None of this is unreasonable, but it is real, and it is the reason RAG should be chosen deliberately.

If the application has a handful of fixed rules, write them in the system prompt or in code. A retrieval pipeline that finds three static sentences is overhead with failure modes. If the authoritative answer is in structured data, such as an inventory count or a price, query the database; embedding a table of prices and asking a model to read them back is slower, costlier, and less reliable than a SELECT. If the entire task concerns one short document the user just uploaded, include that document in the prompt.

Retrieval is also risky when the content is highly sensitive and you cannot enforce document-level access in the search layer. A single shared index with no permission metadata turns every user's question into an unbounded read over everything. In that case the right move is to fix access control first or to not build the feature. And if you cannot show that retrieval improves outcomes on real questions, the added complexity has no justification yet. The Similarity Search and Retrieval Pipeline lessons explain how to measure that.

An internal engineering handbook assistant

The handbook is 4,000 pages across a wiki, changes weekly, and some pages are restricted to managers. Employees ask "How do I request production access?" Pasting everything is impossible and would ignore restrictions. This is a RAG case: chunk and index the pages with their ACLs, retrieve only what the asking employee may read, and cite pages. Freshness lag and permission tests become first-class requirements.

A live bank balance

A customer asks "What is my balance?" The answer is one number in a system of record that changes by the second. Retrieval over documents cannot produce it and an embedding of last night's statement is wrong. Use an authenticated API call executed by trusted code, return the value to the model as a tool result, and have the model phrase it. If the customer also asks what overdraft fees apply, that part can come from retrieved fee policy.

Summarizing one uploaded report

A user uploads a 30-page report and asks for a summary. The report fits in the context window, belongs to one user, and is needed whole. Put it in the prompt. Retrieval would only risk dropping the paragraphs that matter for a summary, because a summary needs coverage, not the top few matches. If the report were 3,000 pages, you would summarize in stages or retrieve per section, but that is a different problem.

1. 1

   Is the needed fact an exact value in a live system?

   Use a deterministic tool. Do not ask a model or a document index to guess it.
2. 2

   Does everything relevant fit comfortably, for every user?

   Put it in the prompt and keep the system simple.
3. 3

   Is it large, changing, permissioned, or must it be cited?

   Use retrieval, and budget for ingestion, evaluation, and access control.
4. 4

   Do you need a style or skill rather than facts?

   Consider fine-tuning, possibly on top of the above.

You now have the reasons for RAG and the boundaries of where it applies. The next lesson draws the full system: how documents are prepared offline, how a question is answered online, and where each failure can occur.