Hallucinations

Watch how a model can produce a polished answer even when the context does not support the claim.

Why do LLMs hallucinate? Because a language model is trained to produce likely text, not to verify claims, and it has no built-in signal for its own uncertainty, so a fluent sentence can be wrong. LLM hallucinations are generated content (a fact, quote, number, identifier, or citation) that the evidence the system had does not support, presented as though it did; they usually appear when the evidence is missing, ambiguous or ignored. This is different from prompt injection, where an attacker plants instructions, and from a truthful answer drawn from an outdated document. You manage hallucinations by supplying trustworthy evidence, checking claims against it, and letting the system abstain or escalate; a citation alone does not guarantee a claim is correct.

An internal HR assistant is asked: "Can contractors claim the $500 home-office stipend?" The retrieval step finds the Employee Benefits Guide v7, whose stipend section covers full-time employees and says nothing about contractors. The assistant replies:

QUESTION : Can contractors claim the $500 home-office stipend?

EVIDENCE : [doc: Benefits Guide v7, sec 4.2]
           "Full-time employees may claim a one-time home-office stipend of $500 within 90 days of start date."

ANSWER   : "Yes. Contractors are eligible for the $500 stipend as long as they submit a receipt within 30 days
            (see Benefits Guide v7, section 4.7, 'Contractor Reimbursements')."

Each sentence is grammatical and confident. Checked against the evidence, the failures are precise: the claim "contractors are eligible" has no supporting text; "within 30 days" contradicts the 90 days the guide states for employees; and "section 4.7, Contractor Reimbursements" does not exist. A reader trusting that citation might file a claim that will be rejected, or worse, make a hiring decision based on it. The rest of this chapter explains why this happens and builds the machinery that would have caught it.

Research and industry use the word differently; some reserve it for statements that contradict the source document, others for any factually false output. For building systems it helps to separate failure types, because each needs a different fix:

• Factual error: the statement is false in the world (a wrong date).
• Unsupported claim: it may or may not be true, but nothing in the provided evidence supports it (the contractor eligibility above).
• Fabricated citation: a reference to a source, page, or section that does not exist, or does not say what is claimed.
• Retrieval failure: the right document was never retrieved, so the model had no chance to be grounded.
• Reasoning error: all facts were correct but the conclusion does not follow (a wrong sum).
• Instruction violation: the model ignored a constraint (answer only from the provided text).

Distinguishing these in your evaluation data is more useful than a single bucket, because "make the model more truthful" is not an engineering task, while "improve retrieval recall for contractor policies" is.

As the Next-Token Prediction lesson shows, training minimises the loss on the text that followed each prefix. If, in the training data, a policy question is followed by a confident "Yes, contractors are eligible…", the model learns that confident, specifically formatted answers follow such questions. It has no internal step that checks whether a specific clause exists. The pattern of "answer + parenthetical section reference" is extremely common in text, so producing one is high-likelihood regardless of whether the section is real. Post-training (Model Alignment) can reward admitting uncertainty, which helps, but it is learned behaviour and imperfect.

Equally, confident tone is not a signal: the model's style comes from learned conventions of helpful writing, not from an assessment of its own knowledge. Token probabilities carry some information (the logprob sketch shows how to read them), but they are an imperfect and uncalibrated proxy, and a fabricated section number can look as probable as a real one.

Each case lists what went into the system, where it failed, what came out, and how you would detect it. The figure places all seven on the request path.

1. Stale knowledge

Input: "What is the current API rate limit?" with no documents. Failure:the model answers from training data that predates a limit change. Output: "100 requests per minute" (it is now 60). Detection: compare against a freshness-dated source of truth; require answers to cite a document with a version and date, and flag answers with no citation.

2. Ambiguous request

Input: "What's the limit for Pro?" (Pro plan or Pro API tier?). Failure:the model silently picks one interpretation. Output: a correct answer to a question that was not asked. Detection: evaluation cases with deliberately ambiguous inputs, expecting a clarifying question.

3. Irrelevant retrieval

Input: the retriever returns a marketing page about "Pro" features. Failure:the model builds an answer from loosely related text. Output: plausible, evidence-shaped, wrong. Detection: measure retrieval recall and relevance separately; check whether cited passages actually contain the claim.

4. Conflicting evidence

Input: two documents (2022 and 2024) give different limits. Failure: the model blends them or picks arbitrarily. Output: "between 60 and 100". Detection:include document dates and precedence rules in the prompt and in the evaluation, and flag answers that cite superseded versions.

5. Unsupported inference

Input: evidence says employees may claim the stipend. Failure: the model infers "and contractors" from the absence of an exclusion. Output: the earlier incident. Detection: claim-to-evidence verification (below) marks the claim "no supporting passage".

6. Fabricated reference

Input: "cite your source". Failure: the model generates a citation-shaped string. Output: "Benefits Guide v7, section 4.7". Detection: deterministic check that every cited ID is in the retrieved set and that the quoted text appears in that document.

7. Tool-output misinterpretation

Input: a tool returns {"balance": 1250, "currency": "cents"}. Failure: the model reads it as dollars. Output: "Your balance is $1,250." Detection: typed tool outputs with units in field names, and tests with known values.

In multi-step systems a hallucination at one step becomes trusted input to the next. Consider a workflow that summarises a thread, calls a tool, then replies. Step through the figure to watch the error travel and see where an interception stops it.

1. 1

   Step 1: summarise the thread

   Thread says "customer asked about extending the trial". The summary model writes "customer is on the Enterprise plan" (the thread never said so; it saw "enterprise-style questions"). Interception point: summaries should list only facts with message references, and store them as claims, not truths.
2. 2

   Step 2: call a tool

   The agent uses the summary to call get_pricing(plan="enterprise"). The tool works fine and returns enterprise pricing. Interception point: the plan should come from the account system via an authenticated lookup, not from model text.
3. 3

   Step 3: final reply

   The model quotes enterprise prices to a customer on a standard plan, citing a real pricing page. Everything the user sees is internally consistent and cited. A reviewer who checks only the last step would find a real document and might approve it.

The lesson: verify at boundaries. Treat each model output that feeds another step as an untrusted proposal, pull identifiers and states from systems of record, and keep intermediate state inspectable in traces so a reviewer can see that step 1 introduced the error.

Grounding means tying output to explicit evidence. The main tools: retrieval from trustworthy, versioned sources (see RAG Architecture); asking for exact passage references; calling structured tools for facts that live in databases; allowing abstention ("I could not find this in the provided documents"); and deterministic policy checks in code.

A citation can exist and still fail. The cited document may be real but not say the claim (a topically related passage); it may say something similar with a different number; the quote may be paraphrased beyond what the text supports; or the model may cite the right document for the wrong claim in a multi-claim sentence. So verifying that a citation resolves is necessary and insufficient: you must verify that the cited passage entails the claim.

Applied to the HR incident, the claim extractor splits the answer into atomic claims, and the verifier compares each with the retrieved passages:

The verifier can be rules plus a second model call acting as an entailment judge, with its own limits (below). Where the verdict is not clearly supported, the system accepts only the supported part and either abstains or escalates the rest.

Build a test set from real questions, each with the evidence a correct system should find, a reference answer (or "should abstain"), and the type of failure it probes. Have two reviewers label independently with written guidelines; adjudicate disagreements with a third reviewer, and record the guideline change if the disagreement exposes an ambiguity. Measure inter-annotator agreement so you know how noisy your labels are.

Define each metric separately, since one number hides which component failed: correctness (is the answer right?), faithfulness (is every claim supported by the provided evidence?), citation accuracy (do cited passages support the attached claims?), retrieval recall (was the necessary evidence retrieved?), abstention appropriateness (does it decline when evidence is insufficient, and answer when sufficient?), and task success. Mind both error types: a stricter verifier produces fewer false negatives (missed unsupported claims) and more false positives (valid answers blocked), and each has a business cost that you should weigh explicitly.

An illustrative set of 20 cases, not a benchmark: 12 answered correctly with supporting evidence; 3 answered with unsupported claims; 2 answered incorrectly despite citing a real document that did not support the claim; and 3 appropriate abstentions. Assume all 17 answered cases included citations, and that the 3 unsupported answers were on questions where the evidence was insufficient.

answered = 12 + 3 + 2 = 17            abstained = 3

Answer correctness (among answered)  = supported_correct / answered      = 12 / 17 = 70.6%
Unsupported-claim rate (answered)    = unsupported / answered            =  3 / 17 = 17.6%
Citation-failure rate (cited answers)= cited_but_unsupported / answered  =  2 / 17 = 11.8%
Abstention precision                 = appropriate_abstentions / abstentions = 3 / 3 = 100%
Abstention recall                    = appropriate_abstentions / cases_needing_abstention
                                     = 3 / (3 abstained + 3 unsupported answers) = 3 / 6 = 50%
Task success (answer right OR abstained appropriately) = (12 + 3) / 20 = 75%

With only 17 answered cases the uncertainty is large: a 95 percent Wilson interval for 12 of 17 is roughly 47 to 87 percent, so differences of a few points between two systems on this set mean nothing. Grow the set until the intervals are narrow enough to support the decision you need to make.

Production inputs drift: documents are revised, products launch, users ask new questions. Track source freshness (age of cited documents, and whether a newer version exists), citation resolution rate, the share of answers that abstain or escalate, and user corrections. Sample answers for human audit each week, stratified by topic and by low-confidence signals, and feed confirmed failures back into the regression set. Route high-stakes domains (legal, medical, financial) to mandatory human review.

An LLM used as a judge can scale checks but has known weaknesses: it may prefer longer or more confident answers, share biases with the model being judged, and be inconsistent between runs. Calibrate any judge against a human-labelled sample, report its agreement, and use it for triage rather than final decisions on critical content. Verification adds latency and cost (a second model call per claim batch), so apply it where the cost of an unsupported claim justifies it, and cache results for repeated documents.

These are easy to confuse because both end in a wrong output. A hallucination is the model producing unsupported content on its own account. Prompt injection is an adversary placing instructions in content the model reads so that it acts against the application's intent. The mitigations differ: grounding and verification for the first; trust separation, least privilege, and deterministic authorisation for the second (see System Prompts vs User Prompts). A poisoned document can cause either: the model may repeat a false claim that was in the document, which is faithful to the evidence yet false, or follow an embedded instruction.

With the verification flow in place, the same question produces a different answer. The claim extractor finds four claims; the verifier keeps the one supported claim and rejects the other three; the system composes only what the evidence supports:

"The Benefits Guide v7 (section 4.2) describes a one-time $500 home-office stipend for full-time employees,
claimable within 90 days of the start date. The guide does not mention contractors, so I can't confirm whether
they are eligible. I've flagged this for the HR benefits team; you can also ask them directly at benefits@company.example."

[trace] claims=4  supported=1  rejected=3 (unsupported, contradicted, unresolved citation)  action=partial_answer+escalate

The answer is less satisfying than "Yes!" and it is correct about what is known. That trade-off, accepting a less confident answer in exchange for being right about its support, is the core design decision in building reliable LLM applications. Next, you will see how weight-level adaptation can complement, but not replace, these controls.

Grounding changes what the model sees. Fine-tuning changes the model itself; the next lesson explains when that is worth it.