Testing CLIs

Test at three levels, and put most of the tests at the cheapest one that can see the behaviour.

The framework-specific in-process runners are shown on the framework pages (Python, Java, Node.js, Go & Rust and Swift & .NET); this page is about what to assert and how to run the black-box layer.

The --json output is a published interface, so test it like one: run a command against fixed input, and compare stdout with a checked-in golden file. When the comparison fails, the change is either a bug or a deliberate interface change, and in the second case the golden file changes in the same commit, where a reviewer sees it (and sees the schemaVersion policy applied).

Four habits keep golden tests useful rather than noisy.

Add a test for the error side of the contract too: on failure with --json, stdout is empty and stderr holds exactly one envelope with schemaVersion, error.code, error.exitCode and a hint.

A prompt that waits for input freezes an agent until its timeout. The rule of TTY, Colour & Interactivity is that a tool prompts only when stdin and stderr are terminals, so the test is simply to run every command that could ask a question with stdin closed and a hard timeout, and to require a fast, informative failure.

Three variations complete the picture: run with DOCSCTL_NO_INPUT=1 and stdin open (a pipe that never writes), to prove the environment variable disables prompts on its own; run with CI=1; and run the interactive path only under a pseudo-terminal in a separate, clearly labelled test. The environment in the example is deliberately minimal so that a developer’s own token or configuration cannot make a test pass.

Exit codes are the cheapest signal an agent has, so each code of the taxonomy gets a test that produces it on purpose. Keep the table in one place and generate the cases from it.

Also test the non-obvious edges: the code for a usage error is the same in every subcommand (frameworks differ here, and the framework pages say how to normalise it); an idempotent no-op returns 0; a dry run returns 0; and a run killed by SIGINT or SIGTERM exits 130 or 143 after cleaning up, which needs a test that starts the process, sends the signal and reads the status.

--help is documentation and, for an agent, the main way to discover flags (see Self-Describing Help), so snapshot it: one snapshot per command, plus the schema --json command tree. A changed snapshot then shows in review that a flag was added, renamed or lost. Make the snapshot independent of the environment first: set the terminal width (for frameworks that wrap help to the width), disable colour with NO_COLOR=1, and strip anything that varies, such as a version string or a home directory.

The command tree from docsctl schema --json is the better snapshot for machines, because it is stable, structured and the same source the MCP wrapper uses (see Documenting CLIs).

bats-core is a test framework for Bash. A test is an @test "description" { …​ } block, and the run helper executes a command and records its exit status in $status and its output in $output (stdout and stderr combined unless --separate-stderr is given, which adds $stderr). run -N command asserts an exit status and run ! command a non-zero one; the tests below check $status explicitly. setup and teardown run around each test, load sources helper files, and bats_require_minimum_version states the minimum Bats version a file needs. Because run executes the real binary with real streams, it is a natural fit for the black-box layer.

The official helper libraries (bats-assert, bats-support, bats-file) add readable assertions. Reference: bats-core documentation.

Click and Typer ship a CliRunner that invokes the command in the test process with given arguments, stdin and environment, and returns a Result with exit_code, output, stdout, stderr and exception (check in the documentation of your version which of these combine the two streams). The Click documentation describes two capture modes: the default replaces sys.stdout and sys.stderr, and the fd mode redirects the file descriptors and catches output from C extensions and logging, but is not available on Windows. Because the runner’s stdin is not a terminal, it is also a cheap way to test the no-prompt rule.

The Click documentation advises using a process-based runner rather than threads to parallelise tests, and notes that isolated_filesystem() is deprecated in favour of temporary directories such as pytest’s tmp_path. See the full example on Python CLI Frameworks. References: Click — Testing Click Applications and Typer — Testing.

picocli lets a test replace the writers of a CommandLine: setOut and setErr take a PrintWriter, and execute returns the exit code, so a JUnit test asserts on the code, on stdout and on stderr separately. The command must write through the writers of its CommandLine (spec.commandLine().getOut() and getErr()) for the capture to see its output; the abbreviated example on the Java page prints to System.out for brevity, and a testable command does not.

The picocli manual distinguishes black-box tests, which invoke the application as a user would and check output and exit code, from white-box tests, which inspect the command objects after parsing; it also points to libraries for testing environment variables and System.err. Run the black-box suite against the native image as well, as described in Packaging & Distribution. Reference: picocli — Testing Your Application.

trycmd runs many CLI cases from files and compares their output with snapshots. A .trycmd or .md file holds the command after a $ ` prompt, the expected output, and an optional `? <status> line for the exit status; [..] and …​ elide the parts that vary. A .toml case gives precise control over stdin, stdout and stderr. Run the cases from one integration test, and refresh them with TRYCMD=overwrite cargo test (TRYCMD=dump writes candidate snapshots to a dump/ directory).

insta is the tool for snapshots of values: assert_snapshot! for text and assert_json_snapshot! (with insta’s json feature) for structured data, with redactions for the fields that vary, cargo insta test to run and cargo insta review to accept or reject changes interactively, and a failing test in CI when a snapshot differs. Together they cover the two halves of the contract: trycmd the exit status and text, insta the --json shape.

References: trycmd and insta. Rust unit, integration and doc tests in general are covered in the Rust Reference’s Testing & Benchmarking.

testscript, from github.com/rogpeppe/go-internal, runs scripts in the txtar format: command lines followed by files marked -- name --. Each script runs in its own temporary directory with a controlled environment. exec runs a program, stdout and stderr match regular expressions against its output, cmp compares files, and a leading ! negates a command, so ! exec requires failure. testscript.Main registers a Go function as a command inside the scripts, which lets the test binary run your main without building it first.

A script asserts failure or success; to pin a specific non-zero status, assert on the exitCode field of the error envelope, as above. Reference: testscript.

Start-up time is part of the interface for a tool an agent calls many times (see Agent Ergonomics), so it deserves a measurement, not an opinion. hyperfine runs a command repeatedly and reports statistics; --warmup N runs first on a warm cache, --runs N sets the number of runs, --shell=none (-N) removes the shell’s own overhead from short commands, --prepare runs a command before each timing, and --export-json and --export-markdown write the results.

Use it to compare builds (JVM against native image, before against after a refactor) on one machine, and treat results from shared CI runners as trends rather than gates, because they are noisy. Do not copy numbers from someone else’s machine into documentation; publish the command. Reference: hyperfine.

Deterministic tests show that the CLI keeps its promises. They cannot show whether a model can use it: whether it finds the right subcommand from --help, reads the error hint, recovers from a usage error and stops at the confirmation step. That is what an agent eval measures: an LLM is given realistic tasks and the CLI, and you observe what happens.

Model output varies between runs, so repeat each task several times and look at the rate rather than one result, and keep evals separate from the merge gate: the deterministic suites gate a release, and evals track quality over time and after every interface change. The general method, including trajectory checks, tool-call accuracy, LLM-as-judge rubrics and regression suites, is on Agent Evaluation; evaluating an agent skill that teaches the CLI is on Testing & Evaluating Skills; how tools should be designed in the first place is on Tool Design. Reference: Anthropic — Writing effective tools for agents (an engineering article).