Testing & Evaluating Skills

A skill’s description decides whether Claude reaches for it at all, before a single line of its body is read — so the first thing worth testing is not the skill’s instructions but its triggering, independently of whether the instructions themselves are correct. Anthropic’s own skill-authoring guidance calls this out as the starting point rather than an afterthought: "Create evaluations BEFORE writing extensive documentation. This ensures your Skill solves real problems rather than documenting imagined ones," and prescribes an evaluation-driven loop — identify gaps by running Claude on representative tasks without the skill, build a handful of scenarios that test those gaps, establish a baseline, write the minimal instructions needed to pass, then iterate — see Anthropic — Skill authoring best practices, "Evaluation and iteration". Its documented evaluation record is a small, data-driven JSON object naming the skill under test, the query, any input files, and the behaviour expected in response:

That page, written for Agent Skills on the Claude API, notes that it has no built-in runner for these records and that users can build their own. Claude Code now ships two runners of its own, covered under Trigger accuracy in Claude Code below. The best-practices checklist for a shippable skill closes with a testing section worth pairing with the record above: "At least three evaluations created," "Tested with Haiku, Sonnet, and Opus," and "Tested with real usage scenarios."

For trigger accuracy specifically, an evaluation record only tells you whether the skill produced the right result once invoked; it says nothing about whether the request should have invoked this skill, a different one, or none at all. Build that out as a small should-fire / shouldn’t-fire table alongside the description before trusting it against neighbouring skills:

To run the table by hand, type each prompt in a fresh session, so leftover context from writing the skill doesn’t mask a weak description, and check which skill, if any, loaded. Both tools also let you force the skill by name, which separates "the instructions work" from "the description triggers":

See Claude Code — Skill not triggering and VS Code — Agent Skills ("Use skills as slash commands").

A skill that scaffolds files, edits a manifest, or runs real tooling (npm install, a test runner, a linter) cannot be trusted from reading its SKILL.md alone — the only way to know its embedded commands and templates actually work against real tooling is to run them. Doing that against the repository the skill lives in risks leaving behind generated files, partial edits, or state a later git status has to untangle by hand. The pinned ai-catalog repository’s own convention is to scaffold and run everything in a throwaway directory outside the repository, keyed off $TMPDIR, so a verification pass can never touch the skill’s own source tree:

Extending the skills and agents catalog restates the same rule for the catalog’s own setup and CI-workflow skills as one of its recurring conventions — "Throwaway-directory verification — exercise every scaffold/workflows/gate template against the real tooling in a directory outside this repository ($TMPDIR/iru-verify/<stack>/<skill>/), fold whatever quirks surface back into the SKILL.md, never delete anything inside this repository to do it, and report whatever couldn’t be verified locally instead of assuming it works." Generalised, the pattern has four parts regardless of which tool runs the skill: a scratch directory under the OS temp path ($TMPDIR, or %TEMP% on Windows) that lives entirely outside the repository under test; running the skill’s actual embedded commands and templates there, not a paraphrase of them; folding back into the skill whatever quirk the real tooling surfaces that the instructions did not anticipate (a flag that moved, an output path that differs from what was assumed); and reporting — in the skill’s own description, a verification note, or a PR — exactly what could and could not be exercised locally, instead of asserting untested behaviour works.

Trigger accuracy answers "did the right skill run"; a golden-output check answers "did it produce the right result" — comparing the actual output of a real run against a known-good reference instead of eyeballing it. Anthropic’s documented evaluation record already carries the raw material for this: its expected_behavior array is a checklist to verify against the transcript of an actual run, not just a description of intent. Where a skill’s output is itself a file rather than free text — a generated CHANGELOG.md entry, a scaffolded build.yml, a rewritten license header — the more exacting version of the same check is a literal golden file: a known-correct example of that output, checked into the skill’s own test fixtures or a throwaway verification directory, that a real run’s output is diffed against. A mismatch either means the skill regressed or the golden file itself needs updating because the correct output genuinely changed — and a reviewer should be able to tell which from the diff.

For a skill packaged in a plugin, claude plugin eval graders cover the same ground without a hand-written script: regex checks the reply or a file for a pattern (or, with match: not_contains, its absence), file_exists checks that the run created a file matching a glob, and baseline has a judge compare the run against a reference transcript:

A regex grader reads Claude’s final reply unless target says otherwise, so this one points at the contents of the scaffolded file. See Claude Code — Test plugins with evals ("Grader types" and "What a grader can look at").

A skill in active use accumulates a body of prompts that once triggered it correctly and outputs that were once right; editing its description or its steps risks breaking either without anyone noticing until it happens in production. Anthropic’s documented iteration workflow makes this explicit by splitting the two Claude instances that catch this kind of drift: "Claude A" drafts and refines the skill, while a fresh "Claude B" — with the skill freshly loaded, no memory of the design discussion — runs it against real tasks and reveals what changed. The prescribed loop after any edit is: "Use the Skill in real workflows …​ Observe Claude B’s behavior …​ Return to Claude A for improvements …​ Apply and test changes …​ Repeat based on usage." In practice this means re-running the should-fire / shouldn’t-fire table above and any golden-output checks after every edit to a skill’s description or steps, not only when the skill is first written — a description edited to fix one weak trigger can just as easily narrow or widen what else it now matches, and a step reworded for clarity can silently change what the skill actually does. Keep a small, named set of representative prompts/fixtures for a skill precisely so re-running them after an edit is mechanical rather than something a reviewer has to reconstruct from memory each time.

claude plugin eval exits non-zero below a score threshold, so the same command can gate a CI job; skill-creator’s version comparison runs a blind A/B between two versions of a skill. See Claude Code — Test plugins with evals and Claude Code — Run evals with skill-creator.

Neither Claude Code’s skills reference nor the Agent Skills open standard defines a dedicated version key for a SKILL.md the way a package manifest does — the frontmatter reference on Skills covers metadata (a free-form map Claude Code stores but does not act on) and license. The Agent Skills specification’s own metadata example records a version there, as data for your own tooling:

See Agent Skills — Specification ("`metadata` field"). A skill shipped in a plugin does get an enforced version: the plugin’s plugin.json version, which pins installed users to that version until it changes (see Plugins & Marketplaces). The Claude API’s own Skills endpoint does version Anthropic-managed skills explicitly ("skill_id": "pptx", "version": "latest", per Anthropic’s Agent Skills quickstart) — but that mechanism belongs to skills uploaded and served through the Skills API, not to a project’s own .claude/skills/ folder read straight off disk. For a project skill, the pinned ai-catalog repository does not version individual skills at all; it versions the repository the skills live in, the same way it versions any other source file:

Generalised, a project that wants finer-grained tracking than "read the git log" can adopt either or both of two conventions neither tool enforces: a ## Changelog heading inside the SKILL.md body itself (or a sibling CHANGELOG.md in the skill’s own folder) noting what changed and why on each edit, and a repository-wide convention — pinning example links and cross-references at a commit SHA the way this sub-section does — so that documentation referencing a skill’s exact behaviour does not silently drift out of sync with a later edit.

Anthropic’s skill-authoring best-practices guide documents an iterative, two-instance workflow for developing and refining a skill, summarised above under regression suites: one instance ("Claude A") helps design and refine a skill’s instructions, while a second, freshly-loaded instance ("Claude B") tests it against real tasks, with observations fed back to the first. The same page notes that Claude understands the Skill format natively, so asking it to create a skill produces a properly structured SKILL.md without a special system prompt.

Claude Code’s skill-creator plugin automates the testing half of that loop. Install it from the official marketplace, then ask for an evaluation of an existing skill:

The plugin walks you through writing test cases and then:

See Claude Code — Run evals with skill-creator. A practical order: draft with one instance (grounded in an existing example skill, per Creating and updating skills and agents's "ground it in an existing example" step), write the evaluation cases before extensive instructions, let skill-creator run them in fresh contexts, and iterate.

The LLMOps & Evaluation sub-section under Guides & References / AI covers evaluation at the level of a whole AI product or pipeline — offline eval suites, LLM-as-judge scoring, production monitoring and drift detection — of which the skill-level trigger-accuracy and golden-output checks on this page are one narrow, project-local instance.