Supply Chain Security
|
This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production. This section’s bibliography lists the reference material consulted while preparing these pages. |
Written on 2026-10-01 against the OWASP Top 10 for LLM Applications 2026 edition (LLM04 Supply Chain), the OWASP Top 10 for Agentic Applications (ASI04 Agentic Supply Chain Vulnerabilities), MITRE ATLAS AML.T0010, NIST AI 600-1 (July 2024), safetensors 0.8.0, model-signing 1.1.1 and CycloneDX 1.7.2 (see the version baseline). No code on this page was run against a live DocsAssistant; the snippets are illustrative and derived from the official documentation linked after each one.
A conventional application depends on libraries, base images and build tools. An LLM application depends on all of those plus artefacts that are opaque to code review: model weights, tokenizers, adapters, datasets, embeddings, prompt templates, and tools that the model is allowed to call. A weights file cannot be read like source code, and a dataset cannot be diffed for intent. The practical response, paraphrased from the supply-chain discussions in Wilson and Albada and the operational chapters in Huyen, is to treat every such artefact like third-party binary code: know where it came from, pin exactly which version you use, verify it before loading, and keep an inventory so you can answer "are we affected?" within minutes of an advisory.
Attack: how the DocsAssistant can be compromised through what it depends on
The DocsAssistant answers questions about this documentation site. Its dependency graph, and what can go wrong at each node, looks as follows.
The recurring attack patterns are these.
| Vector | What happens | Why it works |
|---|---|---|
Malicious serialised model |
A repository ships weights in Python’s pickle format; loading the file executes code chosen by the author. |
Pickle is a program, not a data format. Many PyTorch checkpoints ( |
Look-alike or hijacked repository |
An attacker publishes |
Model names are mutable labels. A bare |
Tampered weights without code execution |
A model that loads safely but has been fine-tuned to leak data on a trigger phrase or to give subtly wrong answers. |
Safe file formats protect against code execution, not against malicious behaviour. This overlaps with data and model poisoning (LLM05). |
Package hallucination ("slopsquatting") |
A coding assistant suggests a plausible but non-existent package name; an attacker registers that name with malicious code and waits for developers to install it. |
Models repeat the same invented names across runs, so the names are predictable. |
Third-party plugins, skills and MCP servers |
A tool server changes its tool descriptions after approval, or a skill carries hidden instructions or scripts. |
Tool metadata is text read by the model, and updates are pulled without review. |
Compromised build inputs |
A base image, a transitive Maven dependency or a CI action is replaced upstream. |
Unpinned tags and version ranges resolve to different content over time. |
Package hallucination has been measured: a USENIX Security 2025 study of code-generating models reported that a substantial share of generated package suggestions did not exist, and that many invented names recurred, which is what makes squatting them profitable. The term "slopsquatting" was popularised by Seth Larson. The exact rates depend on the model and prompt, so treat any single percentage as a snapshot rather than a constant.
Researchers have also reported pickle-based Hub models that evaded the Hub’s automated scanning with deliberately malformed or unusual archives, so scanning reduces risk but is not a guarantee; the safe format and pinning below do not rely on it.
Defend
Prefer safetensors and refuse pickle
The safetensors format stores raw tensors plus a JSON header and contains no executable code, so loading it cannot run arbitrary code. It also supports lazy and memory-mapped loading, which is why most Hub models now publish it. See Model Formats for the format comparison and Conversion and Quantization for converting an existing checkpoint.
from safetensors.torch import load_file
from transformers import AutoModel
# 1. Load raw tensors from a safetensors file: no code can run during loading.
state = load_file("models/docs-embedder/model.safetensors", device="cpu")
# 2. When using transformers, require safetensors and never run repository code.
model = AutoModel.from_pretrained(
"models/docs-embedder",
use_safetensors=True, # fail instead of falling back to a pickle checkpoint
trust_remote_code=False, # do not execute Python shipped inside the repository
)
# 3. If a pickle checkpoint is unavoidable, restrict what torch.load may unpickle.
import torch
weights = torch.load("legacy.pt", map_location="cpu", weights_only=True)
The first snippet derives from the safetensors documentation, the use_safetensors
and trust_remote_code arguments from the Transformers model
documentation, and weights_only from the torch.load
reference. weights_only=True narrows what can be unpickled but the PyTorch documentation still recommends loading only
trusted checkpoints, so convert legacy files to safetensors in an isolated environment rather than relying on it.
Pin every artefact to an immutable revision
A Hub repository is a Git repository. A branch name such as main moves; a commit SHA does not. Resolve the SHA once, review
or scan that exact revision, record it in configuration, and always download by SHA. The mechanics of repositories and
revisions are covered in Hub Essentials.
import os
from huggingface_hub import HfApi, snapshot_download
REPO = "BAAI/bge-small-en-v1.5"
# One-off, in a review step: find out which commit you are about to trust.
info = HfApi().model_info(REPO, revision="main", token=os.environ.get("HF_TOKEN"))
print(info.sha) # copy this value into configuration after review
# At deploy time: download exactly that commit, and only the file types you accept.
PINNED_SHA = os.environ["EMBEDDER_REVISION"] # a full 40-character commit SHA
path = snapshot_download(
repo_id=REPO,
revision=PINNED_SHA,
allow_patterns=["*.safetensors", "*.json", "*.txt"], # excludes *.bin and *.pkl
)
The calls derive from the huggingface_hub download guide and the
HfApi reference. For offline production hosts, download
once in CI, store the directory in an internal artefact repository, and set HF_HUB_OFFLINE=1 at runtime so the service can
never fetch an unreviewed revision (see Running Local LLMs in Production).
Hub datasets and Spaces accept the same revision argument.
Choosing a model is also a supply-chain decision: prefer publishers you can identify, check download history and licence, and read the model card for training data and known limitations (see Choosing Models and Licences). Model and dataset cards are self-reported by the publisher, so they are evidence for a review, not proof; a card that is missing, vague about data sources or lacks an evaluation section is itself a finding.
Scan, but do not depend on scanning
The Hub scans uploads (malware and pickle import scanning, with partner scanners) and flags suspicious files on the repository page. In your own pipeline, scan before a revision is promoted to the internal store, for instance with ModelScan or picklescan for any pickle-based file, and run the load itself in a sandbox without network access or credentials. Scanning looks for known-dangerous imports; it cannot judge behaviour, so it complements the format and pinning rules above.
Keep an inventory: SBOM and ML-BOM
An SBOM lists software components; an ML-BOM (also called AI-BOM) extends it with models, datasets and their provenance.
CycloneDX supports this natively through the machine-learning-model and data component types and a modelCard object.
A minimal, illustrative fragment for the DocsAssistant’s embedding model follows; fields such as hashes would be filled in by
the pipeline that downloaded the artefact.
{
"bomFormat": "CycloneDX",
"specVersion": "1.7",
"components": [
{
"type": "machine-learning-model",
"name": "bge-small-en-v1.5",
"group": "BAAI",
"version": "<pinned commit sha>",
"externalReferences": [
{ "type": "distribution", "url": "https://huggingface.co/BAAI/bge-small-en-v1.5" }
],
"modelCard": {
"modelParameters": { "task": "feature-extraction", "architectureFamily": "bert" },
"considerations": { "technicalLimitations": ["English only"] }
}
}
]
}
The structure derives from the CycloneDX ML-BOM capability page and the CycloneDX specification; check property names against the schema of the version you emit, since the model card fields have changed between releases.
Verify signatures with Sigstore model signing
The OpenSSF model-signing project (1.1.1 at the time of writing) signs a whole model directory using Sigstore: a short-lived
certificate bound to an OIDC identity, a signature over a manifest of file hashes, and an entry in a transparency log. Consumers
verify that the files they hold match what a named identity signed, which detects tampering in transit or in a mirror. It proves
who published the files and that they were not altered; it does not prove the model is benign.
pip install model-signing
# Publisher: sign the model directory with an OIDC identity (opens a browser flow locally).
model_signing sign sigstore ./docs-embedder --signature docs-embedder.sig
# Consumer: verify before loading, naming the identity and issuer you expect.
model_signing verify sigstore ./docs-embedder \
--signature docs-embedder.sig \
--identity "release-bot@example.org" \
--identity_provider "https://accounts.google.com"
The commands follow the model-transparency README and the
Sigstore documentation; flag names have changed between releases, so confirm them with
model_signing --help for the installed version. Verification should be a mandatory step in the pipeline that populates the
internal model store, and the expected identity belongs in reviewed configuration, not in the artefact being verified.
Pin and verify the Java side
The DocsAssistant’s Java backend has the same exposure through Maven dependencies and plugins. Maven Central does not verify a signer for you, so the controls are exact versions, enforced rules and an SBOM. The Enforcer plugin below fails the build on unpinned plugins, SNAPSHOT dependencies and conflicting transitive versions, and the CycloneDX plugin emits the SBOM next to the artefact.
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>${enforcer.version}</version>
<executions>
<execution>
<id>supply-chain-rules</id>
<goals><goal>enforce</goal></goals>
<configuration>
<rules>
<requirePluginVersions/>
<requireReleaseDeps/>
<dependencyConvergence/>
</rules>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.cyclonedx</groupId>
<artifactId>cyclonedx-maven-plugin</artifactId>
<version>${cyclonedx.plugin.version}</version>
<executions>
<execution>
<phase>package</phase>
<goals><goal>makeAggregateBom</goal></goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
The rules derive from the Maven Enforcer built-in rules and the goal from the CycloneDX Maven plugin documentation; the version properties are deliberately left for the project to set to the current releases. Maven has no built-in equivalent of Gradle’s dependency verification file with checksums, so projects that need it combine a private repository manager that proxies and quarantines Central, signed-artefact checks, and the SBOM compared against a vulnerability feed such as OSV or Dependency-Track. Spring AI and LangChain4j version numbers should come from the project’s BOM import, pinned to one reviewed release, rather than from version ranges. The same reasoning applies to container base images: pin by digest and scan as described in Image Best Practices and Security.
Treat package suggestions from models as untrusted
When a coding assistant proposes a dependency, check that the name exists, that its publisher and age are plausible, and that it
is what you meant, before adding it. Prefer lock files with hashes (pip install --require-hashes, uv lock) and an internal
registry mirror that only serves reviewed packages, so a squatted name fails to resolve instead of installing. The
developer-toolchain side of this problem is covered in
Securing AI-Assisted Development; the output-side risk of the model
inventing sources is covered in Misinformation and Hallucination Risk.
Review plugins, skills and MCP servers like dependencies
A third-party MCP server, plugin or skill extends the model’s capabilities and its attack surface at once, and its tool descriptions are read by the model as instructions. Controls that hold up in practice are:
-
Maintain an allow-list of approved servers and skills, each pinned to a version or commit and reviewed before first use.
-
Diff tool names, descriptions and schemas on every update and re-approve changes, because a server can alter them after you approved it (often called a rug pull).
-
Run servers with the least privilege and separate credentials per server; never pass the user’s token through (see MCP Security).
-
Prefer servers you host or whose source you have read; for remote servers, authenticate them and pin their endpoints.
-
Apply the same review to agent skills and plugins that contain scripts, which run with the developer’s permissions.
Agent-level consequences, including cascading effects between tools and agents, are discussed in Excessive Agency and Tool Security and OWASP Top 10 for Agentic Applications, where the supply chain is ASI04.
Test
Supply-chain controls are mostly verified by pipeline checks rather than by attack tooling. Suggested checks, all documented by their upstream projects:
-
Fail the build if any model download lacks a revision pin or resolves to a pickle file (a unit test over the configuration and the downloaded file list).
-
Run ModelScan or picklescan on every candidate artefact and verify the signature before promotion.
-
Regenerate the SBOM on each build and diff it; an unexpected new component or a changed hash blocks the release.
-
Probe the assistant used for coding for invented packages with garak’s
packagehallucinationprobe, which asks a model for code and checks whether the packages it imports exist:
python -m garak --model_type openai --model_name gpt-4o-mini --probes packagehallucination
The command follows the garak reference and expects OPENAI_API_KEY in the environment; probe
names and options can differ between garak versions, so list them with python -m garak --list_probes. No results were
produced against a live DocsAssistant for this page, and none are claimed. For wider scenarios see
AI Red Teaming, and for running these checks continuously see
LLMSecOps.
Map
| OWASP LLM 2026 | OWASP Agentic | MITRE ATLAS | NIST AI 600-1 risk |
|---|---|---|---|
LLM04 Supply Chain |
ASI04 Agentic Supply Chain Vulnerabilities |
AML.T0010 AI Supply Chain Compromise |
Value chain and component integration |
LLM05 Data and Model Poisoning (adjacent: tampered weights and datasets) |
ASI06 Memory and Context Poisoning (adjacent, for poisoned stored context) |
AML.T0020 Poison Training Data |
Information security; information integrity |
The ATLAS technique names are as confirmed on 2026-10-01 in the version baseline; the Agentic names follow the official list on the Agentic applications page.
References
-
Spracklen et al., "We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs", USENIX Security 2025: https://arxiv.org/abs/2406.10279