AI Security & Responsible AI

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This sub-section was written on 2026-10-01 against the OWASP Top 10 for LLM Applications 2026 edition, the OWASP Top 10 for Agentic Applications 2026 (published 2025-12-09), MITRE ATLAS, NIST AI RMF 1.0 with the AI 600-1 Generative AI Profile, the EU AI Act (Regulation (EU) 2024/1689) as amended by the Digital Omnibus on AI, garak 0.17.0, PyRIT 1.1.0, promptfoo 0.123.1, NeMo Guardrails 0.24.1, Guardrails AI 0.11.0, Presidio 2.2.364, LangChain 1.4.3 on Python 3.12 and Spring AI 2.0.1 on Spring Boot 4.1.x with Java 21; the full list is in the version baseline below. Versions and framework contents were read on that date from PyPI, npm, Maven Central, GitHub releases and the publishers' own pages. The code on the pages is illustrative: it is written from the official documentation of each tool and is not compiled by this repository, and the red-team tools were not run against a live DocsAssistant here, so each page says which commands were documented from the official documentation only. No scan results or benchmark numbers are quoted, because none were produced.

The sibling sub-sections show how to build an assistant that works on a good day. This one is about the bad day: a user who types an instruction into a chat box, a web page that was written to be read by a model rather than a person, a tool that does more than the developer intended, and a regulator who asks what you did about it.

Why LLM security is different

Most of what you know about application security still applies: authenticate callers, validate input, patch dependencies, keep secrets out of source control, log what matters. Three properties make LLM applications different enough to need their own reference.

  • Non-determinism. The same input can produce a different output on the next call, and a vendor can change the model under you. A test that passes once proves little, so security tests are statistical: they sample many attempts and report a rate, as AI Red Teaming shows.

  • A natural-language attack surface. In a classic application, code and data are separate channels and an injection is a bug that mixes them. An LLM receives instructions and data in the same channel, as text, and cannot reliably tell which is which. Every document, web page, e-mail or tool result the model reads is a potential instruction. This is why Prompt Injection cannot be fully solved at the model level and is handled by layers around it.

  • Excessive autonomy. Once a model can call tools, an injected instruction stops being a wrong answer and becomes an action: a record deleted, a message sent, money spent. Agency multiplies the consequence of every other weakness, which is why Excessive Agency and Tool Security is as important as the filters.

The practical consequence is a change of stance. Treat the model as an untrusted, persuadable component that sits inside your system, and put the controls around it (least privilege, validation of what goes in and comes out, human approval for consequential actions, budgets, monitoring) rather than expecting the model to police itself. The same stance is called zero trust or pessimistic trust boundaries in the security literature the bibliography lists.

The framework map

Five bodies of work recur on every page. They answer different questions and are meant to be used together.

Five frameworks arranged left to right by question. What can go wrong: the OWASP Top 10 for LLM Applications 2026, LLM01 to LLM10, and the OWASP Top 10 for Agentic Applications, ASI01 to ASI10. How attackers do it: MITRE ATLAS tactics and techniques and the MAESTRO seven-layer agentic threat model, exercised with garak, PyRIT and promptfoo. What you must do: the NIST AI RMF with the AI 600-1 Generative AI Profile, which organises risk management into Govern, Map, Measure and Manage, feeding the EU AI Act, whose obligations are legally binding and dated.
Framework What it is Where it is used here

OWASP Top 10 for LLM Applications (2026)

A community-ranked list of the ten most critical risks for applications built on LLMs.

OWASP Top 10 for LLM Applications and every risk page.

OWASP Top 10 for Agentic Applications (ASI01 to ASI10)

A companion list for autonomous, tool-using agents.

OWASP Top 10 for Agentic Applications.

MITRE ATLAS and CSA MAESTRO

ATLAS is a knowledge base of adversary tactics and techniques against AI systems; MAESTRO is a layered threat-modelling method for agentic systems.

Trust Boundaries and Threat Modeling and the Map step of every risk page.

NIST AI RMF 1.0 and AI 600-1

A voluntary risk-management framework (Govern, Map, Measure, Manage) and its profile for generative AI with twelve named risks.

NIST AI RMF.

EU AI Act

Binding regulation with risk tiers, general-purpose-AI duties and transparency obligations, with dated application.

EU AI Act.

The running scenario: attacking and defending DocsAssistant

The system under test is the DocsAssistant used by the RAG Systems, MCP, Conversational Channels, Voice Agents and LLMOps sub-sections (see RAG Systems in Production): an agent backed by retrieval-augmented generation over this site’s documentation, with five parts that each add attack surface.

  • A search_docs retrieval tool over a vector store of documentation, some of it uploaded by users.

  • An open_issue write tool that files a ticket on the user’s behalf.

  • An MCP server exposing those tools to other hosts (see MCP security).

  • Chat channels such as a web widget, Teams or WhatsApp (see Conversational Channels).

  • A voice channel over telephony (see Voice Agents).

Every risk page repeats the same four steps, so that a reader who has seen one page can skim the next:

  1. Attack. A minimal, defensive demonstration against DocsAssistant. Attacks are shown only as far as needed to understand the defence; there is no catalogue of working jailbreaks.

  2. Defend. The defence in Python (LangChain middleware and guardrails) and in Java (Spring AI advisors and Spring Security), each snippet followed by the link to the official page it derives from.

  3. Test. A garak, PyRIT or promptfoo configuration that would show whether the defence holds.

  4. Map. The risk’s OWASP 2026 ID, MITRE ATLAS technique and NIST AI 600-1 risk, so a finding can be reported in any framework’s vocabulary.

Reading path

Read in order the first time; later, jump to the risk you are facing.

  1. Foundations. Trust Boundaries and Threat Modeling draws the system and the lines an attacker crosses.

  2. The lists. OWASP Top 10 for LLM Applications and OWASP Top 10 for Agentic Applications give the vocabulary and point to the page that treats each risk.

  3. Risks and defences. Prompt Injection, Sensitive Information and Hidden Context, Excessive Agency and Tool Security, Improper Output Handling, RAG and Vector Security, Supply Chain Security, Unbounded Consumption and Denial of Wallet and Misinformation and Hallucination Risk.

  4. Controls and processes. Guardrails, AI Red Teaming, LLMSecOps and Securing AI-Assisted Development.

  5. Governance and regulation. NIST AI RMF, EU AI Act and Responsible AI Practices.

  6. Reference. The Cheat Sheet (PDF) condenses the whole sub-section onto one A4 page.

Security is also covered where it first arises in the sibling sub-sections, which this sub-section links rather than repeats: MCP security, CLI security for agents, Agent guardrails and safety, RAG access control and privacy, RAG guardrails and prompt injection, LangChain guardrails and security and Spring AI security and guardrails.

Version baseline

Every version and status below was read on 2026-10-01. Each page repeats the subset it uses in its lead paragraph; if a page and this table ever disagree after an update, this table wins.

Area Versions and status

OWASP Top 10 for LLM Applications

The 2026 edition is the current one. The OWASP GenAI Security Project resource page shows 2026-08-03 as its publication date and the source repository’s README states 2026-08-04; both are cited here as August 2026. The ordering is LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Excessive Agency, LLM04 Supply Chain, LLM05 Data and Model Poisoning, LLM06 Unbounded Consumption, LLM07 Misinformation, LLM08 Hidden Context Exposure (replacing System Prompt Leakage), LLM09 Vector and Embedding Weaknesses and LLM10 Improper Output Handling. The previous edition (2025) is kept in the repository’s 2025/ directory, and the older v1.1 (2023) edition is the one used by The Developer’s Playbook. The mapping between the three editions is on its own page.

OWASP Top 10 for Agentic Applications

2026 edition, published 2025-12-09 by the OWASP GenAI Security Project: ASI01 Agent Goal Hijack, ASI02 Tool Misuse and Exploitation, ASI03 Identity and Privilege Abuse, ASI04 Agentic Supply Chain Vulnerabilities, ASI05 Unexpected Code Execution (RCE), ASI06 Memory and Context Poisoning, ASI07 Insecure Inter-Agent Communication, ASI08 Cascading Failures, ASI09 Human-Agent Trust Exploitation and ASI10 Rogue Agents. Names checked against the official PDF ("Version 2026", December 2025) on 2026-10-01.

MITRE ATLAS and MAESTRO

ATLAS is read from the mitre-atlas/atlas-data repository. The technique IDs used in this sub-section were confirmed on 2026-10-01: AML.T0051 LLM Prompt Injection, AML.T0054 LLM Jailbreak, AML.T0020 Poison Training Data, AML.T0010 AI Supply Chain Compromise, AML.T0024 Exfiltration via AI Inference API, AML.T0029 Denial of AI Service, AML.T0056 Extract LLM System Prompt, AML.T0057 LLM Data Leakage, AML.T0070 RAG Poisoning, AML.T0053 AI Agent Tool Invocation. MAESTRO is the Cloud Security Alliance’s layered threat-modelling framework for agentic AI (blog post of 2025-02-06).

NIST

AI RMF 1.0 (NIST AI 100-1) and the Generative AI Profile NIST AI 600-1 (July 2024), which names twelve risks: CBRN information or capabilities, confabulation, dangerous, violent or hateful content, data privacy, environmental impacts, harmful bias and homogenization, human-AI configuration, information integrity, information security, intellectual property, obscene, degrading or abusive content, and value chain and component integration.

EU AI Act

Regulation (EU) 2024/1689. Prohibited practices have applied since 2025-02-02 and general-purpose AI obligations since 2025-08-02. The Digital Omnibus on AI, Regulation (EU) 2026/1744, in force since 2026-07-27, moves the Annex III high-risk obligations to 2027-12-02 and the Annex I (product-embedded) high-risk obligations to 2028-08-02. The Commission’s AI Act implementation timeline page lists the same two dates, shows Article 50 transparency obligations applying from August 2026 and lists a new prohibition on non-consensual intimate content from December 2026. Article 50(2) marking obligations apply from 2026-12-02 for systems already on the market. This sub-section is not legal advice; the dates are re-stated with their sources on the EU AI Act page.

Red-team tools

garak 0.17.0 (NVIDIA/garak, released 2026-09-09), PyRIT 1.1.0 (microsoft/PyRIT, released 2026-09-04; the Azure/PyRIT repository linked by older material is archived), promptfoo 0.123.1 on npm.

Guardrail tools

Llama Guard 4 (12B) and Llama Prompt Guard 2 (86M) model cards on Hugging Face, ShieldGemma (google/shieldgemma-2b), NeMo Guardrails 0.24.1, Guardrails AI 0.11.0, Presidio 2.2.364. LangChain 1.4.3 provides PIIMiddleware, HumanInTheLoopMiddleware, ModelCallLimitMiddleware and ToolCallLimitMiddleware; Spring AI 2.0.1 provides SafeGuardAdvisor and moderation models such as OpenAiModerationModel.

Supply-chain tools

safetensors 0.8.0, Sigstore model-signing 1.1.1 (PyPI, released 2025-10-10, the sigstore/model-transparency project), CycloneDX specification 1.7.2 with its ML-BOM capability, cyclonedx-python-lib 11.12.0.

Frameworks

LangChain 1.4.3 and langchain-core 1.6.6 on Python 3.12; Spring AI 2.0.1 on Spring Boot 4.1.x with Java 21.

Where the books are out of date

The requester-provided books are used for concepts only; where they differ from current material, the pages follow the official documentation:

  • OWASP list. The Developer’s Playbook (Wilson) follows version 1.1 of the OWASP Top 10 for LLM Applications from 2023. Two newer editions exist (2025 and 2026), with different names and numbering, so this sub-section cites 2026 as current and maps the older identifiers.

  • Agentic threats. The agentic list post-dates the books, and the MCP-specific threats (tool poisoning, confused deputy, token passthrough) are only partly covered by Building Applications with AI Agents (Albada).

  • Regulation. The EU AI Act timeline changed after the books were written; the pages use the dated, sourced version.

  • Commercial products. Vendor and product names in the books may have changed owners or names, so the pages name only open-source tools in depth.

  • Frameworks. Spring AI in Action (Walls) targets Spring AI 1.0; the pages use the 2.0.1 advisor and moderation APIs.

Bibliography

The following sources were consulted while preparing the pages of this sub-section. Every code example was written from the current official documentation of the tool it uses — never copied from a book. All links below were requested on 2026-10-01 and resolved, except the oreilly.com publisher pages and the oreil.ly supplement link, which answer automated fetches with an HTTP 403 response although they are live pages in a browser, and the EUR-Lex link, which answers automated fetches with an HTTP 202 placeholder response.

Requester-provided books

Their text and code are not reproduced; concepts are paraphrased and credited on the page that uses them. No book PDF is committed or attached to this repository.

Official documentation

OWASP

Threat modelling

Models and platforms

Supply chain

Guardrails

Red teaming

Framework-specific controls

Specifications and standards

Papers and engineering articles

These books are consulted references, not the primary source of any example anywhere in this section; on any discrepancy the official documentation of the tool or framework in question is authoritative.

References