Misinformation and Hallucination Risk

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against the OWASP Top 10 for LLM Applications 2026 edition (LLM07 Misinformation), NIST AI 600-1 (July 2024), LangChain 1.4.3 on Python 3.12, Spring AI 2.0.1 on Java 21, garak 0.17.0 and promptfoo 0.123.1 (see the version baseline). The snippets are illustrative and were not compiled or run here.

A model that invents a fact is not attacked by anyone; the weakness is inherent to how language models generate text, which is explained in Hallucinations and Limitations. It becomes a security and liability problem when an application presents the invention as authoritative, when a user or a downstream system acts on it, and when an attacker can steer it, for example by publishing the package name a coding assistant is known to invent. This page treats misinformation as a risk with owners, controls and tests. The detection techniques themselves (claim extraction, NLI checks, self-consistency, citation verification) are covered in Hallucination Detection and Correction and are not repeated.

Why it counts as a security risk

Three properties distinguish an operational misinformation risk from a quality bug.

  • Fluency hides the error. Wrong answers use the same tone and structure as right ones, so users cannot calibrate from style.

  • Consequences land on the operator. Courts and tribunals have so far treated what an organisation’s own chatbot or filing says as the organisation’s responsibility (see the cases below).

  • Errors can be induced or exploited. Poisoned documents (see RAG and Vector Security), ambiguous prompts and predictable fabricated names are all levers an adversary can pull.

Documented cases

Only cases with a primary source are listed. Summaries are paraphrased; read the linked decisions before relying on a detail.

  • Fabricated case citations in a court filing. In Mata v. Avianca, Inc. (S.D.N.Y., No. 1:22-cv-01461), lawyers filed a brief containing cases that a chatbot had produced and that did not exist. In an opinion and order dated 2023-06-22 the court imposed sanctions, with the finding that the lawyers continued to stand by the citations after doubts were raised. The lesson is that the professional remains accountable for verifying output, and that asking the same model "is this case real?" is not verification.

  • A support chatbot stating a refund policy that did not exist. In Moffatt v. Air Canada, 2024 BCCRT 149, the British Columbia Civil Resolution Tribunal held the airline liable for incorrect bereavement-fare information given by its website chatbot and rejected the argument that the chatbot was a separate entity responsible for its own statements. This is a first-instance tribunal decision in one jurisdiction, so treat it as an illustration of the risk rather than settled law everywhere.

  • Defamation claims about fabricated allegations. Claims have been filed over model statements about named individuals (for example Walters v. OpenAI in Georgia, where the defendant obtained summary judgment in 2025). Outcomes depend on jurisdiction and facts and the case history is still developing, so this page cites it only as evidence that the claim type exists; check the current docket before drawing conclusions.

The common thread for the DocsAssistant: an answer such as "the retryPolicy option defaults to three attempts" that is simply wrong costs a customer a production incident, and the vendor’s published assistant said it.

Attack: how misinformation is produced and exploited

Minimal and defensive; no payloads beyond what a test needs.

  • Unprompted fabrication. A user asks about a feature that does not exist ("how do I enable the batch-export flag?") and the model describes it fluently.

  • Retrieval gaps. The retriever returns near-miss chunks, and the model fills the gap from parametric memory while sounding grounded. See the failure analysis in Hallucination Detection and Correction.

  • Sycophancy and leading questions. A false premise in the question ("since version 4 removed the old API, how do I…​") is accepted and elaborated.

  • Poisoned sources. An editable wiki page or uploaded document asserts something false; the model repeats it with a citation, which makes it more convincing (OWASP LLM05 and LLM09 on the related pages).

  • Package and API hallucination. A coding assistant invents a dependency or method name; see the dedicated section below.

flowchart LR Q["User question"] --> R["Retrieve sources"] R --> G{"Sources support an answer?"} G -- "yes" --> A["Answer with citations"] G -- "no" --> X["Abstain or escalate"] A --> V{"Citations verified?"} V -- "yes" --> O["Show answer plus disclosure"] V -- "no" --> X X --> O

The diagram is the target design for the controls below: the model is never the last word on whether it knows something.

Defend

The defences are layered. None removes hallucination; together they reduce its frequency and, more importantly, its impact.

Grounding and citation requirements

Require the answer to come from supplied sources and to point at them. Make the citation machine-checkable: each claim cites chunk identifiers that the application itself issued, and the application rejects identifiers it did not issue. A simple measure of how much of an answer is supported is the groundedness ratio

\[ G = \frac{|S_{\rm supported}|}{|S|} \]

where \( S \) is the set of claims (or sentences) in the answer and \( S_{\rm supported} \) those entailed by the cited sources. A release gate such as \( G \ge 0.95 \) on an evaluation set is a policy choice to be made per product, not a standard. How to estimate \( S_{\rm supported} \) is the subject of the RAG page linked above.

The Python sketch below uses LangChain’s structured output to force a citation and abstention field, then validates the citations in ordinary code. It follows the structured-output and agent pages of the LangChain documentation; confirm the response_format and result key names against your installed version.

from pydantic import BaseModel, Field
from langchain.agents import create_agent

class Answer(BaseModel):
    answer: str = Field(description="Answer using only the provided sources")
    citations: list[str] = Field(description="Chunk ids from the sources")
    can_answer: bool = Field(description="False if the sources do not contain the answer")

agent = create_agent(
    model="openai:gpt-4o-mini",
    tools=[],
    system_prompt=(
        "Answer only from the numbered sources provided by the user. "
        "If they do not contain the answer, set can_answer to false."
    ),
    response_format=Answer,
)

def ask(question: str, sources: dict[str, str]) -> str:
    context = "\n".join(f"[{cid}] {text}" for cid, text in sources.items())
    result = agent.invoke({"messages": [("user", f"{context}\n\nQuestion: {question}")]})
    out: Answer = result["structured_response"]
    if not out.can_answer or not out.citations or not set(out.citations) <= set(sources):
        return "I could not find this in the documentation."
    return out.answer

This derives from the LangChain documentation on structured output and agents; the model name is an example and the API key is read by the provider integration from the environment. The check that every cited identifier was issued by the application is the part that carries the security value, because it cannot be talked out of its decision.

The Java equivalent maps the response to a record with Spring AI’s structured output and applies the same check.

public record Answer(String answer, List<String> citations, boolean canAnswer) {}

public String ask(ChatClient chatClient, String question, Map<String, String> sources) {
    String context = sources.entrySet().stream()
            .map(e -> "[" + e.getKey() + "] " + e.getValue())
            .collect(Collectors.joining("\n"));

    Answer out = chatClient.prompt()
            .system("Answer only from the numbered sources. If they do not contain the answer, set canAnswer to false.")
            .user(context + "\n\nQuestion: " + question)
            .call()
            .entity(Answer.class);

    if (!out.canAnswer() || out.citations().isEmpty()
            || !sources.keySet().containsAll(out.citations())) {
        return "I could not find this in the documentation.";
    }
    return out.answer();
}

This derives from the Spring AI reference pages on structured output and ChatClient. In a real service the check belongs in a reusable advisor (see Spring AI Advisors) or, in LangChain, in middleware (see LangChain Middleware); Spring AI also ships evaluator classes for relevancy and fact checking that are suited to offline tests, but verify their current names and signatures in the reference before use.

Abstention

"I don’t know" must be a first-class, rewarded output. Practical rules: prefer an explicit can_answer field to free text, set a minimum retrieval score below which the model is not called at all, route abstentions to a human or a search link, and measure the abstention rate. A rate near zero on questions outside the corpus is a defect, and a rate that is too high is a usability defect; track both. Escalation patterns are in Human in the Loop.

Domain limitation

Narrowing what the assistant is willing to discuss reduces the surface on which it can be wrong. The RAISE framework in Wilson’s The Developer’s Playbook for Large Language Model Security (see the bibliography) begins with limiting the application’s domain; the paraphrase here is that an assistant should be scoped to the tasks and subjects it was built and evaluated for, and should decline the rest. For the DocsAssistant that means documentation questions only: no legal, medical or pricing commitments, and no speculation about unreleased features. Implement the scope in three places: the system prompt, a topic classifier or guardrail at input (see Guardrails), and the corpus itself, because a model cannot cite what was never indexed. Prompt-only scoping can be bypassed, so it is a usability measure, not a boundary (see Prompt Injection).

User education and disclosure

  • Say it is AI. Label generated answers clearly. Some jurisdictions require this for chatbots; see EU AI Act for the transparency obligations and their dates.

  • Show the evidence. Render citations as links to the source passage so a user can check in one click.

  • State limits where decisions happen. A short line such as "verify before using in production" next to code and configuration answers, not buried in terms of service. The Air Canada decision suggests that a disclaimer on a different page does not reliably shield the operator, so disclosure complements controls rather than replacing them.

  • Train reviewers. Anyone who accepts model output on behalf of the organisation (support agents, lawyers, developers) needs to know the failure modes and the verification step expected of them. See Responsible AI Practices.

Package and API hallucination in coding assistants

Coding assistants sometimes recommend dependencies or methods that do not exist. Research by Spracklen et al. (USENIX Security 2025) measured package hallucination across many models and found it common and, for a given prompt, often repeatable, which is what makes it exploitable: an attacker registers the invented name with malicious content and waits. The practice is widely called slopsquatting. Controls:

  • resolve every new dependency against an internal allow-list or proxy repository, and require a human to approve new names;

  • check that the package exists, has a history, and matches the intended project before installing, and pin versions with hashes in lock files;

  • run the build and tests in a sandbox without production credentials (see Securing AI-Assisted Development);

  • verify invented methods by compiling and testing, not by asking the model again.

The wider dependency story is on Supply Chain Security.

Test

garak, PyRIT and promptfoo can probe for fabrication. The commands below come from their official documentation; no results were produced against a live DocsAssistant, so none are shown. Check the option names against the installed versions.

# garak: probe for hallucinated package names, then list the available hallucination-related probes
python -m garak --target_type openai --target_name gpt-4o-mini --probes packagehallucination
python -m garak --list_probes | grep -i -e hallucin -e snowball

This derives from the garak repository and its reference documentation; the OpenAI generator reads OPENAI_API_KEY from the environment.

# promptfooconfig.yaml (excerpt)
targets:
  - id: https
    config:
      url: ${DOCS_ASSISTANT_URL}
redteam:
  purpose: "Answers questions about the product documentation only"
  plugins:
    - hallucination
    - overreliance
npx promptfoo@latest redteam run

This derives from the promptfoo hallucination plugin and the red teaming guide. For PyRIT, the project documentation describes scorers (for example true/false scorers driven by an LLM) that can classify whether a response asserts something the supplied source does not; the exact classes change between releases, so follow the current PyRIT documentation rather than a snippet from this page. Results from these tools are a sample, not a guarantee; keep a curated regression set of questions with known answers and known non-answers, as described in AI Red Teaming.

Map

OWASP LLM 2026 OWASP Agentic MITRE ATLAS NIST AI 600-1

LLM07 Misinformation (related: LLM05 Data and Model Poisoning, LLM09 Vector and Embedding Weaknesses)

ASI09 Human-Agent Trust Exploitation (an agent’s confident output persuading a human); ASI08 Cascading Failures (a wrong output consumed by another agent)

No technique maps directly to hallucination as an inherent behaviour. Related, not equivalent: AML.T0070 RAG Poisoning and AML.T0010 AI Supply Chain Compromise for induced misinformation and package squatting.

Confabulation; Information Integrity

The Agentic IDs follow the official names recorded in the version baseline. The ATLAS statement is deliberate: ATLAS catalogues adversary techniques, and spontaneous fabrication has no adversary.

References