Hallucinations & Limitations

A large language model is trained to predict the next token that is plausible given what came before, not to look up a fact and report whether it is true. Nothing in that objective distinguishes "the most likely continuation of this text" from "the correct answer" — the two coincide often enough, on well-represented topics, that the distinction is easy to miss until they diverge. When a prompt asks about something the model saw rarely or never during training — an obscure API, a recent event, a made-up citation format — the model still produces a fluent, confident-sounding continuation, because fluency is what next-token prediction optimises for. That continuation can be entirely fabricated while reading exactly like a fact the model actually knows.

Two structural pressures make this worse in practice. First, decoding always produces some token at every step (see Sampling & Decoding): there is no built-in "I have no basis for continuing this" output, so a model asked a question it cannot ground in anything will still complete the answer shape it was asked for. Second, instruction-tuned models are trained in part to be helpful and to comply with the request as asked, which rewards producing an answer over producing no answer, unless the prompt or the training explicitly makes refusal or hedging an acceptable, rewarded response.

A closely related failure mode is inconsistency: asking a model the same question twice, or asking it to verify its own previous answer, can produce two different answers, neither of which is guaranteed to be correct. This follows directly from sampling being probabilistic (Sampling & Decoding covers why even temperature 0 is not fully deterministic): a model that has no single correct answer memorised is sampling from a distribution over plausible continuations, and different samples can land on different, mutually exclusive claims. Inconsistency is also a diagnostic signal, not just a failure: running the same prompt multiple times and comparing the outputs (best-of-N / self-consistency checking, covered with a worked example in Prompting Techniques) is a useful mitigation precisely because agreement across independent samples is evidence the answer is grounded in something stable, while disagreement flags a likely hallucination.

Every model is trained on data collected up to some date, after which it has no direct knowledge of the world — a new library release, a changed API, an event that happened afterward. Asked about anything past that cutoff, a model that is not told about the gap will often still answer fluently, extrapolating from older patterns rather than reporting that it does not know. This is a version of the same underlying problem as general hallucination (confident completion standing in for grounded knowledge), specific to time: the fix is never to trust a model’s unaided claim about "the latest version of X" or "current pricing," and instead to either supply current information in the prompt (grounding, below) or route the question to a tool that can look it up, which is why every dated fact on this page’s own site links to a live vendor page rather than stating a number from memory.

Sycophancy is a model’s tendency to shift its stated position toward whatever it infers the user wants to hear, rather than toward what is actually correct — agreeing with a flawed argument because the user phrased it confidently, or reversing a previously correct answer just because the user pushed back, without the user giving any new evidence to justify the reversal. It is a hallucination in the sense that the model asserts something as true that is not supported by the evidence in front of it, but the trigger is social (the shape of the conversation) rather than a gap in training data. Because sycophancy specifically follows conversational pressure, closed-choice framing and citation-backed verification (both below) help by giving the model a fixed target to check its answer against, one that a user’s disagreement in the next turn cannot silently move.

Package hallucination is a specific, high-impact case of fabrication: a model asked to write code confidently names a library, package or import path that does not exist, because a plausible-sounding package name is, by construction, exactly the kind of fluent continuation next-token prediction produces. The practical danger is that an attacker can register a package under a name a model hallucinates often enough, so that a developer who copies generated code and runs pip install/npm install on the suggested name installs malicious code instead. This risk, and the broader question of validating and sandboxing AI-generated code before it runs, is covered in depth in Misinformation and Hallucination Risk and Supply Chain Security; this page only names the failure mode as one more instance of the general pattern above.

None of the following techniques eliminates hallucination outright — all of them reduce its frequency and, more importantly, make the remaining cases detectable instead of silent.