Improper Output Handling

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

Written on 2026-10-01 against the OWASP Top 10 for LLM Applications 2026 edition (LLM10 Improper Output Handling), MITRE ATLAS (technique ID confirmed on that date, see the version baseline), NIST AI 600-1 (July 2024), LangChain 1.4.3, Spring AI 2.0.1 and Spring Security 6.x. The code is illustrative and was written from the official documentation linked after each listing; no test results were produced against a live DocsAssistant.

Improper output handling is the failure to validate, sanitise or encode what a model produces before it reaches a browser, an interpreter, a database, a file system or another service. The principle is old: output from a component you do not control is untrusted input to the next component. What is new is that the model sits in the middle of a channel where an attacker can write (through a prompt, a retrieved document or a tool result), so the "untrusted" label applies even when the model is your own and well behaved. The OWASP list makes this the tenth risk, but in practice it is the risk that turns a successful prompt injection (see Prompt Injection) into data theft or code execution, so it deserves the same weight as the first.

Attack: what untrusted output can do

The DocsAssistant answers questions about internal documentation in a web widget and in chat channels, and can query a reporting database and open issues. Assume an uploaded document contains hidden text asking the model to include particular content in its answer. The model complies, and the answer is now attacker-authored text flowing into a trusted renderer. The common sinks are summarised below.

Sink What the generated text can do DocsAssistant example

HTML in a browser (XSS)

Script or event handlers execute with the origin’s session.

The widget inserts the reply with innerHTML; an img tag with an onerror handler runs in the user’s session.

Markdown rendered to HTML

Raw HTML passes through, javascript: links run on click, images are fetched automatically.

A reply contains a Markdown image whose URL carries conversation data in its query string.

SQL

A generated query or fragment alters or reads more than intended.

A "text to SQL" report tool runs whatever the model wrote, including a statement that reads another tenant’s rows.

Shell and code interpreters

Generated commands or code run with the service’s authority.

A tool pipes a generated filename into a shell command.

Server-side fetchers

A generated URL makes the server request internal addresses (SSRF).

A "summarise this link" tool fetches http://169.254.169.254/ from inside the cluster.

Downstream systems and templates

Output is concatenated into e-mail, templates, logs or prompts of another agent.

A reply containing template syntax is rendered by a server-side template engine.

The Markdown image case is worth spelling out because it needs no script execution at all. If the interface renders Markdown images, the browser requests the image URL as soon as the reply is displayed. A reply that includes an image whose address is an attacker’s host, with a secret or a summary of the conversation in the query string, has exfiltrated data through the user’s own browser. MITRE ATLAS files data leaving through the model’s output channel under AML.T0024 (Exfiltration via AI Inference API), and the same leak can be carried by links, form targets and unfurled previews in chat channels. The mechanism is described in the security literature on LLM applications (Wilson, Albada) and in public write-ups of exfiltration against commercial assistants; this page shows only the defence.

flowchart LR A["Attacker text in document or message"] --> M["Model"] M --> O["Generated output"] O --> V{"Validate, encode, sanitise"} V -->|"allowed"| R["Renderer or interpreter"] V -->|"rejected"| F["Fallback or refusal"] R --> U["User, database, shell or fetcher"] classDef untrusted fill:#fde2e2,stroke:#a33,color:#000 class A,O untrusted

The diagram restates the rule used throughout this sub-section: the check sits between the model and every consumer, whatever the upstream guardrails did. Which consumers exist is the output of the threat model.

Defend

The defences are the ones web security already prescribes, applied at the model boundary. In order of preference: do not give the model the capability, constrain the output to a schema, then encode or sanitise for the specific sink.

Constrain the shape: schema-validated structured output

When the output feeds code, ask for a structure rather than free text and validate it. A schema reduces the space of possible outputs, but it is not a security control by itself: a field can still contain a hostile string, so values are checked again at the sink. The mechanics of structured output are covered in LangChain structured output, Spring AI structured output and Structured Output and Tool Calling.

from typing import Literal
from urllib.parse import urlparse
from pydantic import BaseModel, Field, field_validator

ALLOWED_HOSTS = {"docs.example.com"}

class Answer(BaseModel):
    text: str = Field(max_length=4000)
    intent: Literal["answer", "refuse", "open_issue"]
    source_urls: list[str] = Field(default_factory=list, max_length=5)

    @field_validator("source_urls")
    @classmethod
    def only_allowed_hosts(cls, urls: list[str]) -> list[str]:
        for url in urls:
            parsed = urlparse(url)
            if parsed.scheme != "https" or parsed.hostname not in ALLOWED_HOSTS:
                raise ValueError("source URL outside the allow-list")
        return urls

# raises pydantic.ValidationError if the model output does not conform
answer = Answer.model_validate_json(model_output_json)

The listing uses Pydantic models as described in the Pydantic validators documentation; LangChain accepts such a model as the response format of an agent, and the LangChain structured output page documents how. The exact LangChain parameter names change between releases, so the snippet validates the JSON explicitly and leaves the wiring to the linked page.

Encode for the sink

Sanitising a string "for safety" in the abstract does not work, because the dangerous characters depend on the destination. Use the encoder for the context: HTML body, HTML attribute, URL, SQL parameter, shell argument. For HTML, either render the reply as plain text (textContent) or render Markdown through a library that does not pass raw HTML and then sanitise the result.

import { marked } from "marked";
import DOMPurify from "dompurify";

const ALLOWED_TAGS = ["p", "ul", "ol", "li", "strong", "em", "code", "pre", "a", "blockquote"];

export function renderReply(markdown, container) {
  const dirty = marked.parse(markdown, { async: false });
  const clean = DOMPurify.sanitize(dirty, {
    ALLOWED_TAGS,
    ALLOWED_ATTR: ["href"],        // no src, style or on* attributes
    ALLOWED_URI_REGEXP: /^https:/i // https links only
  });
  container.innerHTML = clean;     // images are not in the allow-list
}

The configuration options come from the DOMPurify README; the Markdown parser’s own documentation warns that it does not sanitise its output, which is why the second step is mandatory. Dropping img from the allow-list closes the image exfiltration channel; if images are required, rewrite their URLs to a first-party proxy that only serves allow-listed hosts. Add a Content Security Policy as a second layer, for example default-src 'self'; img-src 'self' without unsafe-inline, so that a sanitiser bypass still cannot load remote images or run injected script (see MDN: Content Security Policy).

Never build queries, commands or URLs from model text

  • SQL: give the model a tool that takes typed parameters and runs a prepared statement, not a tool that accepts a SQL string. If free-form SQL is unavoidable, use a read-only database role restricted to the reporting schema, row-level security keyed to the caller, a statement timeout and a parser-based allow-list of statement types.

  • Commands: pass arguments as an array to an API that does not invoke a shell, validate them against a pattern, and run the process in a sandbox with no credentials.

  • URLs: resolve the host, reject loopback, link-local, private and metadata addresses, pin the resolved address for the actual request, disallow redirects to other hosts and apply an allow-list of schemes and domains. This is the standard SSRF defence documented in the OWASP SSRF Prevention Cheat Sheet.

import ipaddress
import socket
from urllib.parse import urlparse

def is_safe_url(url: str, allowed_hosts: set[str]) -> bool:
    parsed = urlparse(url)
    if parsed.scheme != "https" or parsed.hostname not in allowed_hosts:
        return False
    for info in socket.getaddrinfo(parsed.hostname, 443, proto=socket.IPPROTO_TCP):
        address = ipaddress.ip_address(info[4][0])
        if (address.is_private or address.is_loopback
                or address.is_link_local or address.is_reserved):
            return False
    return True

The checks follow the OWASP SSRF Prevention Cheat Sheet; a check followed by a separate request is open to DNS rebinding, so production code connects to the validated address itself, which the snippet does not show.

Python: enforce it in the agent loop

In LangChain 1.x the place for a cross-cutting check on model output is middleware, the same mechanism as the built-in PIIMiddleware and HumanInTheLoopMiddleware (see LangChain Middleware and LangChain Guardrails and Security). A custom middleware can run after the model call, validate the message and replace it with a refusal when the check fails. Hook names and signatures differ between minor versions, so consult the LangChain middleware documentation before writing one; the validation function above is the part that carries the security logic and is independent of the hook. Tool arguments deserve the same treatment as answers: validate them with the tool’s argument schema, and require human approval for write actions (see Human in the Loop).

Java: advisors and the web layer

In Spring AI 2.x, an advisor around the ChatClient is the equivalent hook: a call advisor can inspect the ChatClientResponse and reject or rewrite it (see Spring AI Advisors and Spring AI Security and Guardrails). For typed results, entity(Class) binds the response to a record through the BeanOutputConverter; bind to a record and validate it with Bean Validation before use.

public record Answer(
        @NotBlank @Size(max = 4000) String text,
        @NotNull Intent intent,
        @Size(max = 5) List<@Pattern(regexp = "^https://docs\\.example\\.com/.*$") String> sourceUrls) {

    public enum Intent { ANSWER, REFUSE, OPEN_ISSUE }
}

Answer answer = chatClient.prompt()
        .user(question)
        .call()
        .entity(Answer.class);

Set<ConstraintViolation<Answer>> violations = validator.validate(answer);
if (!violations.isEmpty()) {
    throw new IllegalStateException("Model output rejected: " + violations.size() + " violation(s)");
}

The structured-output binding is described on the Spring AI Structured Output Converter page and the constraints on the Jakarta Bean Validation site. The listing is conceptual: the validator bean and exception handling are omitted, and the exact advisor interfaces should be taken from the Spring AI advisors documentation. On the web side, Spring Security can set the Content Security Policy header through http.headers(h → h.contentSecurityPolicy(…​)), and server-side HTML should be encoded with the OWASP Java Encoder (Encode.forHtml, Encode.forHtmlAttribute) or a template engine with auto-escaping left on. Run the services that execute generated queries under a database role with least privilege, as the threat model requires.

Safe rendering in chat channels

Each channel has its own renderer and therefore its own rules, covered in the Conversational Channels section: Chat UI Protocols explains what the protocol lets the server send, Embedding a Web Chat Widget covers the origin, iframe and CSP decisions of a widget, and Operating Chat Channels covers moderation and monitoring. The common rules are these.

  • Send plain text where the channel allows it; where formatting is needed, convert to the channel’s restricted markup rather than passing HTML through.

  • Disable automatic link unfurling and image previews for model-authored links, or route them through an allow-list proxy.

  • Show the real destination of links, and strip or mark links whose host is not on the allow-list.

  • Render in a sandboxed iframe with a restrictive sandbox attribute when the widget must display rich content.

  • Log the raw model output next to what was rendered so that a bypass can be investigated without reproducing it.

Test

Run output-handling tests as part of the red-teaming programme described in AI Red Teaming; the checks that run on every change belong in the pipeline of LLMSecOps. Three open-source scanners cover this area. The commands below come from their official documentation; the probe and plugin names should be confirmed against the installed version, since they change between releases.

# garak: list probes, then run the ones about markdown exfiltration and web injection
python -m garak --list_probes
python -m garak --target_type openai --target_name gpt-4o-mini --probes xss

# promptfoo: scaffold a red-team configuration and run it
npx promptfoo@latest redteam init
npx promptfoo@latest redteam run

The garak invocation follows the garak documentation (the xss probe family includes a Markdown image exfiltration probe in recent releases; verify with --list_probes) and the promptfoo commands the promptfoo red team quickstart. PyRIT is a framework rather than a command line: its documentation at https://microsoft.github.io/PyRIT/ shows how to define a target and an attack, and an output-handling test there amounts to sending the injection payloads and scoring whether the reply contains an active URL, tag or query. The OpenAI credentials come from the OPENAI_API_KEY environment variable in all three tools. No results were produced for this page against a live DocsAssistant, so none are reported.

Conventional application-security tests remain the most reliable for this risk: unit tests that feed known hostile strings (script tags, javascript: links, image URLs with query strings, SQL fragments, internal IP addresses) through the renderer and the sink wrappers, a browser test that asserts the CSP blocks a remote image, and a DAST scan of the widget.

Map

OWASP LLM 2026 OWASP Agentic (ASI) MITRE ATLAS NIST AI 600-1 risk

LLM10 Improper Output Handling

ASI05 Unexpected Code Execution (RCE), ASI02 Tool Misuse and Exploitation (when generated arguments reach tools and interpreters)

AML.T0024 Exfiltration via AI Inference API (Markdown image and link exfiltration); AML.T0051 LLM Prompt Injection as the usual trigger

Information security

The ASI names follow the official list on the agentic list page. Related risks are LLM01 (the trigger), LLM02 (what leaks) and LLM03 (what the output can then do); the layered view is in Guardrails.

References