Sensitive Information and Hidden Context

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against the OWASP Top 10 for LLM Applications 2026 edition (LLM02 Sensitive Information Disclosure and LLM08 Hidden Context Exposure), MITRE ATLAS (techniques confirmed on that date, see the version baseline), NIST AI 600-1 (July 2024), Presidio 2.2.364, LangChain 1.4.3 and Spring AI 2.0.1. The snippets are illustrative and written from the official documentation; no result of running them against a live DocsAssistant is reported, because none was produced.

Two risks are treated together because they share one rule: whatever reaches the model’s context can come back out. LLM02 is about data the application should not reveal to this particular reader: personal data, customer records, proprietary documents and content memorised during training. LLM08 is about content the developer meant to keep out of sight, such as the system prompt, tool schemas, memory and injected context. In the 2025 edition the second risk was named System Prompt Leakage; the 2026 edition widened it, and a book written against the 2023 v1.1 list (Wilson) will not have either. The sections below paraphrase the data-protection chapters of the books in the bibliography (Wilson, Albada, Huyen, Aryan, Mendelevitch and Bao) and combine them with the current public frameworks.

Where sensitive data leaks

Channel	How it leaks	First defence
Training and fine-tuning data	A model trained or fine-tuned on tickets, e-mails or code can reproduce fragments of them when prompted, especially records that were repeated or are unusual. This is memorisation, and it cannot be patched after the fact.	Do not train on data you could not publish to every user; deduplicate and scrub before training.
Retrieval (RAG over-sharing)	The retriever returns a chunk the asker may not read, and the model summarises it. The model has no notion of who is asking.	Per-document access control in the retrieval filter (see Access Control and Privacy).
Prompts sent to hosted models	Users paste customer data, logs and credentials into the chat. The text then sits with a third party, in its logs and possibly in its retention window.	Classify first, redact before sending, choose the endpoint by classification.
Conversation memory and logs	Memory written for one user is read for another; traces, evaluation datasets and analytics keep prompts and completions in clear text.	Scope memory by authenticated identity; redact before logging; set retention.
Tool results	A CRM or database tool returns a whole row when the answer needs one field, and the whole row enters the context.	Return the minimum; filter columns in the tool, not in the prompt.
The context itself (LLM08)	System prompts, tool names and parameter schemas, memory entries and retrieved text are extracted by a user who asks the right way.	Assume it will be read; keep secrets and authorisation logic out of it.

Channel

How it leaks

First defence

Training and fine-tuning data

A model trained or fine-tuned on tickets, e-mails or code can reproduce fragments of them when prompted, especially records that were repeated or are unusual. This is memorisation, and it cannot be patched after the fact.

Do not train on data you could not publish to every user; deduplicate and scrub before training.

Retrieval (RAG over-sharing)

The retriever returns a chunk the asker may not read, and the model summarises it. The model has no notion of who is asking.

Per-document access control in the retrieval filter (see Access Control and Privacy).

Prompts sent to hosted models

Users paste customer data, logs and credentials into the chat. The text then sits with a third party, in its logs and possibly in its retention window.

Classify first, redact before sending, choose the endpoint by classification.

Conversation memory and logs

Memory written for one user is read for another; traces, evaluation datasets and analytics keep prompts and completions in clear text.

Scope memory by authenticated identity; redact before logging; set retention.

Tool results

A CRM or database tool returns a whole row when the answer needs one field, and the whole row enters the context.

Return the minimum; filter columns in the tool, not in the prompt.

The context itself (LLM08)

System prompts, tool names and parameter schemas, memory entries and retrieved text are extracted by a user who asks the right way.

Assume it will be read; keep secrets and authorisation logic out of it.

The diagram shows the DocsAssistant path with the places where a control belongs. The point is that redaction and access control sit before the model, because after the model has seen a value, no filter on the output is a guarantee.

flowchart LR U["User message"] --> C["Classify and detect PII"] C -->|restricted| L["Local model"] C -->|redacted| H["Hosted model"] R["Retriever with ACL filter"] --> H R --> L S["System prompt: no secrets"] --> H H --> O["Output check"] L --> O O --> A["Answer"] H -.-> G["Logs and traces: redact first"] L -.-> G

Attack

This is a defensive description of what to test for, not a recipe. Three scenarios against the DocsAssistant, each mapped to a technique in MITRE ATLAS.

Data leakage through the answer (AML.T0057, LLM Data Leakage). A user asks about "the migration guide" and the retriever returns a chunk from an internal incident report in the same index. The model answers helpfully with a customer’s name and e-mail address. Nothing was "hacked": the access check was simply missing in the retrieval step.
Extracting hidden context (AML.T0056, Extract LLM System Prompt). A user asks the assistant to repeat its instructions, to translate them, to summarise the tools it can call, or to continue "the text above". The system prompt, the tool names and parameter descriptions come back. If the prompt contained an internal URL, a service-account name or a key, the attacker now has it. Extraction is not a bug in the model; instructions are text in the same channel as the question.
Exfiltration via the inference API (AML.T0024). An attacker with a legitimate account issues many structured queries and assembles memorised or proprietary content, or the model’s behaviour, from the responses. The signal is volume and systematic variation, which rate limits and monitoring are meant to catch.

Memorisation is the training-time version of the first scenario. Research on extracting verbatim training sequences from language models (Carlini and co-authors, see the references) shows that larger models and repeated sequences are more exposed, which is why the defence is upstream: data selection and deduplication rather than output filtering.

Defend

The controls fall into five groups, in order of strength.

Do not put the secret there. A system prompt must be written as if it will be read, so it holds behaviour and tone, never credentials, internal hostnames, or "if the user says X, grant Y" authorisation logic. Authorisation belongs in code. Keys come from environment variables or a secret store (see Key Vault and Secrets) and are used by tools, never passed through the model.
Classify before you send. Decide per data class which endpoint may see it. A simple scheme is enough: public, internal, confidential, restricted. Public and most internal text may go to a hosted model under a contract that fits your policy; restricted data (health, payment, credentials, customer lists) stays on infrastructure you control.
Minimise and redact. Detect personal data in user input, tool results and documents, and replace it with typed placeholders before the model call. Keep the mapping outside the model if you need to restore values in the answer.
Scope what is retrieved and remembered. Filter by the caller’s identity in the retriever and in memory, as described in RAG and Vector Security.
Check and log carefully. Scan outputs for patterns that must never leave (keys, national identifiers) and plant a canary in the system prompt to detect verbatim leaks. Redact before anything is stored in logs, traces or evaluation sets.

PII detection and redaction in Python

Microsoft Presidio is an open-source framework with an analyzer that finds entities such as names, e-mail addresses and card numbers, and an anonymizer that replaces them. It combines pattern matching, checksums and NLP models, so recall is high but not perfect; treat it as a risk reduction, not a guarantee.

Detecting and replacing PII with Presidio before a hosted model call

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import OperatorConfig

analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()

def redact(text: str) -> str:
    findings = analyzer.analyze(text=text, language="en")
    result = anonymizer.anonymize(
        text=text,
        analyzer_results=findings,
        operators={
            "DEFAULT": OperatorConfig("replace", {"new_value": "<PII>"}),
            "EMAIL_ADDRESS": OperatorConfig("replace", {"new_value": "<EMAIL>"}),
        },
    )
    return result.text

print(redact("Contact Ana Lopez at ana@example.com about invoice 42"))

The snippet follows the quick start on the Presidio documentation; Presidio also needs a spaCy language model installed, which its installation page describes, and the default recognisers are English-oriented.

Inside an agent the same idea is available as middleware, so the redaction happens at the model boundary and cannot be forgotten by one code path.

PIIMiddleware on input, output and tool results

from langchain.agents import create_agent
from langchain.agents.middleware import PIIMiddleware

agent = create_agent(
    model="openai:gpt-4.1",
    tools=[search_docs],
    system_prompt="You answer questions about the product documentation.",
    middleware=[
        PIIMiddleware("email", strategy="redact", apply_to_input=True, apply_to_tool_results=True),
        PIIMiddleware("credit_card", strategy="block"),
    ],
)

This is derived from the PII detection section of the LangChain guardrails documentation. The model name is an example and the provider key is read from OPENAI_API_KEY in the environment; the strategies (redact, mask, hash, block) and flags are explained with a tested example on LangChain Guardrails and Security, and the middleware mechanism on LangChain Middleware.

Java: redact before the call and scope by identity

Spring AI does not ship a PII redactor, so the usual pattern is a small component in front of the ChatClient, or a custom advisor that does the same work, with Spring Security supplying the identity used for retrieval and memory. The advisor API is covered on Advisors; the sketch below stays conceptual and keeps the regular expression deliberately small, since a production system should call a real detector (a Presidio service, or a managed equivalent).

A redaction step in front of the ChatClient, with the caller from Spring Security

@Component
class PiiRedactor {
    private static final Pattern EMAIL = Pattern.compile("[\\w.+-]+@[\\w-]+\\.[\\w.]+");

    String redact(String text) {
        return EMAIL.matcher(text).replaceAll("<EMAIL>");
    }
}

@RestController
class DocsController {
    private final ChatClient chat;
    private final PiiRedactor redactor;

    DocsController(ChatClient.Builder builder, PiiRedactor redactor) {
        this.chat = builder.build();
        this.redactor = redactor;
    }

    @PostMapping("/api/ask")
    String ask(@RequestBody String question, Authentication caller) {
        return chat.prompt()
                .system("Answer from the product documentation only.")
                .user(redactor.redact(question))
                .advisors(a -> a.param(ChatMemory.CONVERSATION_ID, caller.getName()))
                .call()
                .content();
    }
}

The ChatClient calls follow the Spring AI ChatClient API page and the memory scoping follows its chat memory page; the exact constant used for the conversation identifier has changed between Spring AI releases, so check it against the version you use. The full per-user pattern, including a canary word that detects verbatim system prompt leaks, is on Spring AI Security and Guardrails.

Data classification and local models

Classification decides routing. The rule of thumb used in the DocsAssistant: a message that the detector tags as restricted is answered by a model running on the organisation’s own hardware, and is never sent to a hosted API, even redacted, because redaction can miss values. Running local models in production, with their sizing and operations, is covered on Running Local LLMs in Production. A local model removes the third-party disclosure path but not the others: it still memorises what it was fine-tuned on, it can still be asked for its system prompt, and its logs still hold prompts.

Hosted providers differ in retention, training-on-your-data defaults and regional processing, and those terms change, so read the current data-processing terms of the provider you use rather than relying on a table here. Regulatory duties (for example under the GDPR) are discussed on EU AI Act and Responsible AI Practices.

Layered guardrails

Input and output guardrails that detect PII or secrets are one layer; see Guardrails for how the layers combine and why none of them replaces the controls above. Secrets that already leaked into a prompt, log or repository must be rotated, not merely deleted; the repository’s secret scanning practice is described under Securing AI-Assisted Development.

Test

Each tool below is documented by its maintainers; the commands are shown as the documentation describes them, and flags change between releases, so run --help or the current docs first. No results are reported here: none of these were run against a live DocsAssistant.

garak (NVIDIA) is a vulnerability scanner for LLMs. Its probes include ones that test whether a model replays training data and whether it can be induced to emit sensitive strings. An invocation has the form python -m garak --target_type openai --target_name <model> --probes leakreplay, with the key supplied in OPENAI_API_KEY. See the garak repository and its probe reference.
promptfoo has red-team plugins for PII exposure, prompt extraction and document exfiltration from RAG. A configuration lists the plugins under redteam: and runs with npx promptfoo@latest redteam run. See the promptfoo red teaming documentation for the current plugin names.
PyRIT (Microsoft) is a framework for building automated red-team attacks, including multi-turn conversations that try to elicit a system prompt or private data. It is a library rather than a one-line scanner; start from the PyRIT repository documentation.

Beyond scanners, three tests belong in the normal test suite: a retrieval test that asks as user A for a document only user B may read and asserts it is not returned; a unit test for the redaction function with a table of positive and negative examples; and a check that a canary string planted in the system prompt never appears in any answer. Combine them with the practices on AI Red Teaming.

Map

Topic	OWASP LLM 2026	OWASP Agentic	MITRE ATLAS	NIST AI 600-1 risk
Personal data, proprietary content or memorised training data in answers	LLM02 Sensitive Information Disclosure	ASI03 Identity and Privilege Abuse where agents hold credentials	AML.T0057 LLM Data Leakage	Data Privacy
System prompt, tool schema or memory extracted	LLM08 Hidden Context Exposure	not a direct match	AML.T0056 Extract LLM System Prompt	Information Security
Bulk extraction through the inference API	LLM02, LLM06 Unbounded Consumption	not a direct match	AML.T0024 Exfiltration via AI Inference API	Information Security, Data Privacy
Over-broad retrieval	LLM02, LLM09 Vector and Embedding Weaknesses	ASI06 Memory and Context Poisoning is the related integrity risk	AML.T0057	Data Privacy

Topic

OWASP LLM 2026

OWASP Agentic

MITRE ATLAS

NIST AI 600-1 risk

Personal data, proprietary content or memorised training data in answers

LLM02 Sensitive Information Disclosure

ASI03 Identity and Privilege Abuse where agents hold credentials

AML.T0057 LLM Data Leakage

Data Privacy

System prompt, tool schema or memory extracted

LLM08 Hidden Context Exposure

not a direct match

AML.T0056 Extract LLM System Prompt

Information Security

Bulk extraction through the inference API

LLM02, LLM06 Unbounded Consumption

not a direct match

AML.T0024 Exfiltration via AI Inference API

Information Security, Data Privacy

Over-broad retrieval

LLM02, LLM09 Vector and Embedding Weaknesses

ASI06 Memory and Context Poisoning is the related integrity risk

AML.T0057