Performance and token budget
| This documentation was generated with the assistance of AI. Please report any inaccuracies. |
Every installed skill costs tokens before it is used: its name and description sit in the skill listing of every
session, and its body is loaded in full each time it is invoked. This page records how large the catalog was, the
budgets that now keep it small, the model assigned to each kind of skill, and the decisions taken while optimizing it
(issue #11). Everything here was changed without altering what a skill writes, reports or asks, with one deliberate
exception: iru-plan and iru-code gained the acceptance-criteria coverage table and the persisted
[NEEDS CLARIFICATION] markers described in Common workflows.
The numbers come from tools/catalog-audit.py (see tools/README.md); the "before" column is the committed snapshot in
tools/baseline/ (catalog-audit-before.md, sample-runs-before.md), and the "after" column comes from the same script
and the same sample runs on the finished work (catalog-audit-after.md, sample-runs-after.md).
Before and after
Static audit totals (python3 tools/catalog-audit.py --markdown; the snapshot of the original tree is
tools/baseline/catalog-audit-before.md):
| Metric | Before | After |
|---|---|---|
Skills / agents |
196 / 3 |
196 / 3 |
Skill description characters (total) |
494,826 |
73,851 |
Agent description characters (total) |
5,092 |
1,459 |
Descriptions over 1,024 characters (skills and agents) |
189 |
0 |
Descriptions over 1,536 characters (the listing cap, so cut off) |
167 |
0 |
Frontmatter that fails a strict YAML parse (loads with no description and no model) |
39 |
0 |
Skills whose frontmatter |
7 |
0 (all pass) |
|
6,217,277 |
5,303,911 |
|
86,433 |
70,353 |
|
39 / 11 |
36 / 0 |
Bytes inside fenced code blocks of |
997,720 |
489,768 |
Supporting files (templates and references) / bytes |
302 / 4,290,070 |
524 / 4,851,168 |
Redundant bytes in duplicate supporting files |
364,713 |
365,397 (not lower: +684 bytes) |
Models: haiku / sonnet / opus (skills) |
85 / 108 / 3 |
58 / 135 / 3 |
Agents with a pinned |
0 of 3 |
3 of 3 (two |
Inline cross-model |
21 |
17 (all accepted) |
Sample runs (claude -p --output-format json --permission-mode acceptEdits, Claude Code 2.1.260; the commands, fixtures and
caveats are in tools/baseline/sample-runs-before.md). Prompt-side tokens are input + cache read + cache creation:
| Sample | What it measures | Prompt-side tokens before | Output tokens before | Cost (USD) before | Duration before | After (tokens / cost / duration) | Output identical? |
|---|---|---|---|---|---|---|---|
S0 |
|
42,787 |
4 |
0.3327 |
5.1 s |
30,278 tokens / 0.2076 / 1.8 s (12,509 fewer prompt-side tokens, -29.2 %) |
n/a |
S1 |
|
1,158,903 |
5,977 |
0.2993 |
69.0 s |
336,756 / 0.3932 / 73.6 s, on Sonnet instead of Haiku, and the run stopped to ask a question without writing files; the same prompt with the model forced back to Haiku: 1,598,692 / 0.3357 / 132.7 s (higher, it now follows the whole documented job) |
Templates byte-identical (14 of 14 skills). Generated files not byte-identical: the model’s own choices differ (integration branch, optional steps), see |
S2 |
|
303,946 |
3,786 |
0.5287 |
58.1 s |
315,447 / 0.0999 / 31.8 s, on Haiku instead of Opus (the old frontmatter did not parse); it stopped to ask for module names. With the model line removed: 298,954 / 0.5744 / 76.0 s |
Yes: the root |
S3 |
|
259,764 |
3,835 |
0.6432 |
59.6 s |
257,500 / 0.6860 / 83.5 s (tokens slightly lower, cost and duration higher) |
Same |
Cost is only comparable between the before and after run of the same sample (the samples use different models and different cache states). Where an after number is not lower, the table says so rather than hiding it.
Read the sample table as follows. S0 is the clean measurement: the fixed prompt shrank by 12,509 tokens (29.2 %), all of it
in the cache-creation part, because the skill listing is smaller. S1 and S2 are not like for like: the model of both skills
changed on purpose (iru-setup-typescript-github-workflows went from Haiku to Sonnet; iru-setup-android-gitignore
ran on the default Opus because its old frontmatter did not parse, and now runs on Haiku), so the cost columns mostly
reflect the model. Where the after number is not lower: S1 and S3 cost, S3 duration and output, the S2 prompt side, the
redundant duplicate bytes (+684), and S1 on the same model (Haiku), where a faithful run reads the templates and writes the
whole file set, so it is more expensive than the old run that skipped parts of the skill. Plugin mode produced the same
S1 files as directory mode (sync_versions.py differs by a two-line guard) once the headless run was allowed to read the
plugin directory (--add-dir); details, every run and the S4 results (/iru-plan coverage table and
[NEEDS CLARIFICATION] marker, /iru-code stopping at its gate) are in tools/baseline/sample-runs-after.md.
Budgets and why
python3 tools/catalog-audit.py --check enforces the budgets in tools/catalog-budgets.json and exits non-zero on any
error. CI runs it on every pull request (.github/workflows/catalog-check.yml).
| Budget | Level | Why |
|---|---|---|
|
error |
Claude Code caps each entry of the skill listing at 1,536 characters and gives the whole listing about 1 % of the context window. With 196 skills the listing cannot hold long descriptions: skills that do not fit are listed by name only, so Claude cannot tell when to use them. |
The invocation form ( |
error |
The part of a description that survives truncation must say how the skill is called. |
Internal skills (only ever invoked by other catalog skills through |
warning |
Nobody types them, so their description only has to be unambiguous to the calling skill. |
|
error |
The whole body is read on every invocation. Large generated-file templates live in |
|
warning |
The recommended size for a skill body. |
Frontmatter parses with a strict YAML parser and |
error |
An unquoted `: ` inside a description made 39 skills fail to parse; Claude Code then loads the skill with no description and no model. |
Every agent pins a |
error |
An unpinned |
Shared reference files stay byte-identical |
error |
They are copied per skill so each skill stays self-contained. |
|
error |
Moving content out of the body must not leave dangling or orphaned files. |
Model tiers
Each skill and agent carries one model. The tier follows the kind of work, not the language:
| Model | Used for | Skills now |
|---|---|---|
|
Mechanical work that runs a tool and reports: test, coverage and code-quality gates, version bumps, |
58 |
|
Writing code and files: every |
135 |
|
Judgment over an unfamiliar repository or an open-ended review: |
3 |
Agents: iru-gate-runner and iru-change-summarizer are haiku; iru-isolated-skill-executor is inherit, because it
runs a whole coding skill whose own model must decide. Before this change about 30 iru-setup-* scaffold and workflow
skills were haiku while 32 comparable ones were sonnet, and no agent set a model.
Prompt-cache analysis
When a skill with a model that differs from the current one is invoked inline through the Skill tool, the model
switches for the rest of the turn. The prompt cache belongs to one model, so the conversation prefix the new model sees
is not cached and is paid for again as a cache write; returning to the first model later can find its entry expired.
A single inline detour is cheap, but a coding skill that ran a haiku gate inline several times per task group would
alternate models repeatedly.
The catalog avoids this in two ways:
-
Gates are not run inline.
iru-code,iru-code-one-task-groupand every<key>-code-one-task-groupskill dispatch each test, coverage, quality, license and security run to theiru-gate-runneragent. A subagent has its own context, its own (small) prefix and its own model, so the caller’s cache is untouched, and only a compact summary comes back into the caller’s context. Whole coding skills run throughiru-isolated-skill-executor, which inherits the model. One accepted exception remains inside the gate runner: the 11 doc-comment audit skills (iru-java-javadoc,iru-kotlin-kdoc,iru-android-dokka,iru-kmp-dokka,iru-dotnet-docfx,iru-typescript-tsdoc,iru-php-phpdoc,iru-python-docstrings,iru-swift-docc,iru-rust-rustdoc,iru-go-godoc) aresonnet, because they write the missing doc comments, and every group skill dispatches them through thehaikuiru-gate-runner, so the model still switches once inside that subagent. The switch stays out of the caller’s context and cache, and the subagent’s prefix is small, which is why it is accepted rather than fixed; the audit does not report it because the call sits inside anAgent({…})prompt. -
catalog-audit.pyreports everySkill()call whose caller and callee models differ and that is not delegated to an agent (the "inline cross-model calls" section), so a new one cannot appear unnoticed. A call counts as delegated when it sits inside anAgent({…})prompt or within 250 characters after a mention ofiru-gate-runneroriru-isolated-skill-executor: the*-generate-all-testsskills describe the dispatch in a sentence and write theSkill()call as the prompt it carries.
The audit found 21 such calls in the original tree. Four of them (the *-generate-all-tests skills of android, dotnet,
kmp and swift, which already run the test skill through iru-gate-runner) are recognised as delegated by the rule
above and no longer reported. The 13 call sites that remain (12 caller/callee pairs) were all reviewed and are recorded
with their reason under accepted_cross_model_calls in the budgets file:
-
10 are calls to
iru-explore(opus) from skills that need its report (project type, technology stack, docs location) in their own context; an agent would return only a summary, and the exploration is a one-off per run. -
2 are
iru-issuecallingiru-planandiru-pr-review(opus): both are interactive (questions, confirmation before posting comments), so they must run in the orchestrator’s conversation, not in an isolated agent. -
1 is
iru-knowledge-compactcallingiru-knowledge-complete(haiku), a single-purpose file operation whose stop message the caller reads; an agent per item would add a context for a few file edits.
The remaining differences of the original list (four *-repository orchestrators that were haiku calling sonnet
scaffolds) disappeared when those orchestrators moved to sonnet.
Splitting the catalog
Ask (5) of the issue was to evaluate splitting the catalog by purpose or by technology. The marketplace implements a split
by technology: iru-core (25 skills, the stack-independent pipeline and the three agents), iru-knowledge (7) and one
plugin per stack, from iru-nestjs (3 skills) to iru-swift (16). The alternative, a split by purpose, was evaluated and
not taken:
-
By purpose would give a
setupplugin (70 scaffold and workflow skills, 26,401 description characters), agatesplugin (59 test, coverage, quality, doc-audit, version-bump and test-generation skills, 19,422 characters) and acodeplugin (39-code-one-taskand-code-one-task-groupskills, 14,360 characters) besides the core. A repository uses one or two stacks, so every one of those plugins would still carry the other sixteen stacks' skills: installing "gates" for a Java library lists 59 skills of which 5 apply. The per-purpose split also crosses the dependency graph the wrong way: a stack’s code skill calls that stack’s gates and scaffolds, so the three purpose plugins would depend on each other and install together in practice. -
By technology matches how the skills call each other: every
Skill()edge is inside a stack or from a stack toiru-core(or to a sibling the stack really reuses, such asiru-springboottoiru-java), which is what letstools/generate-marketplace.pyderive the plugin dependencies from the audit’s call graph and fail on an uncovered edge.
The effect on the skill listing is the point of the split. The listing is the part of the fixed prompt that grows with the number of installed skills (every description, capped per entry, inside about 1 % of the context window):
| Installed | Skills listed | Description characters |
|---|---|---|
Everything (copying the whole |
196 |
73,851 |
|
39 |
15,750 |
|
40 |
16,297 |
|
54 |
21,967 |
So a single-stack repository that installs its plugin carries about a fifth of the listing a full copy would, roughly 4,000 tokens of the 30,278-token fixed prompt measured in S0 instead of the full listing’s share. The split does not change what any skill does: the plugins select directories from the unchanged tree, and copying a skill directory keeps working.
Additional agents: evaluated, not adopted
Candidates considered were role agents such as a planner, an implementer, a reviewer and a documentation writer, in addition to the three existing ones. They were not added:
-
The three agents already cover the shapes that repeat (about 184 gate runs, 101 whole-skill runs and 2 history summaries across the skills). A fourth agent adds another context window and another system prompt to every invocation.
-
The roles of the issue-to-PR cycle are interactive. A subagent returns one summary and cannot hold a dialogue with the user, and
iru-plan,iru-pr-reviewand the ticket skills ask questions. -
The model for each role is already chosen per skill (
opusfor exploration, planning and review,sonnetfor coding,haikufor gates). An agent pins one model, so it would duplicate that choice and remove the ability to vary it by task.
Plugin-mode agent dispatch
A spike (tools/spikes/plugin-resolution.md, Claude Code 2.1.260) showed that when the catalog is installed as plugins:
-
Skill({skill: "iru-java-test"})by bare name still resolves, as does the namespacediru-java:iru-java-test, so none of the 298Skill()call sites needed to change. -
Agent({subagent_type: "iru-gate-runner"})by bare name does not resolve. The agent exists only asiru-core:iru-gate-runner, and the error lists the namespaced names as the available ones. -
.claude/skillsdoes not exist, which is why key discovery first reads the skills available in the session and usesfind .claude/skillsonly as a corroborating check. -
Agents listed on a marketplace entry whose
sourceis the repository root load even thoughclaude plugin detailsshowsAgents (0).
Rewriting about 290 call sites would have broken the copy-a-directory installation, so every skill that dispatches an
agent itself (iru-code, iru-code-one-task-group, iru-issue, iru-release, iru-setup-changelog,
iru-setup-repository and the -repository orchestrators, every -code-one-task-group and *-generate-all-tests
skill, and the few others that use iru-isolated-skill-executor or iru-gate-runner directly) states the rule once at
the top of its body: dispatch as subagent_type: "<agent-name>" when installed under .claude/agents/, or
"iru-core:<agent-name>" when installed as a plugin, using whichever name appears in your available agent types. Each
agent’s body repeats the same line as documentation; the dispatching caller never reads it, which is why the rule has to
live in the callers.
Documented, not done
These were considered and left as options because they change more than this work was approved to change:
-
Consolidating the shared reference files. About 357 KB of the supporting files are byte-identical copies (the UI contract in 12 skills, the CLI contract in 4, and so on). One copy per plugin would remove most of it, but every skill is deliberately self-contained so that copying its directory into another repository keeps working. The copies are instead checked for drift.
-
Merging the per-language gate skills. One cross-language test, one coverage and one quality skill would shrink the listing by about thirty entries, but each skill’s body is specialized per toolchain, the names are part of the public interface, and the gate runner and every code skill refer to them.
-
A physical
plugins/<name>/skills/tree. The marketplace instead lists each plugin’s skill directories over the unchanged.claude/skills/tree (source: "./"). The trade-off is that each installed plugin caches a full copy of the repository (231 MB measured for two plugins from a directory marketplace, see Plugins); a physical tree would remove that but move 196 directories and change the path used when copying a skill, so it needs an explicit decision.
See Plugins and Packaging and distributing skills, agents, and CLAUDE.md for installing the catalog, and Catalog architecture for how the skills compose.