Shells, Completions & Shell Integration

Completion is a feature of the shell, not of the program being typed. When the user presses Tab, the shell has the text of the line so far and nothing else; it does not start shelf to ask what it accepts unless a completion script tells it to. A completion script is therefore a small program written in the shell’s own language, loaded into the interactive shell, which receives the words typed so far and hands back candidates. There are two ways to produce the candidate list:

Because the script runs inside the user’s interactive shell, it must be quick (the user is waiting), must not print anything to the terminal except candidates, and must not assume a particular working directory or an installed helper. A dynamic helper that is slow, or that writes a log line to stdout, breaks the prompt of every user who installed the script.

Bash implements programmable completion with the builtins complete, compgen and compopt. complete registers a specification for a command name, compgen generates candidates (it is the usual building block inside a completion function) and compopt changes options of the completion in progress. The most common specification is complete -F function command, which runs a shell function when Tab is pressed on that command’s line. The function communicates through variables: COMP_WORDS (the words on the line), COMP_CWORD (the index of the word under the cursor), COMP_LINE and COMP_POINT (the raw line and the cursor offset) and the result array COMPREPLY, which the function fills with the candidates. Bash also passes the command name, the word being completed and the previous word as $1, $2 and $3. complete -W "words" offers a fixed word list without a function, and complete -C command runs an external command and uses its output as candidates.

Calling the function with a simulated line on bash 3.2.57 (the helper subcommand was a stub that prints two ISBNs) printed:

Source: GNU Bash Reference Manual — Programmable Completion and Programmable Completion Builtins

The plain $(compgen ...) form splits its output on whitespace, so it breaks for candidates that contain spaces (a book title, for example). The bash-completion project provides helper functions such as _init_completion and _filedir that handle word breaking and quoting; they are available only when that package is loaded, so a script meant to work everywhere either avoids them or tests for them first. complete -o nospace stops bash adding a space after a candidate (for --format= style values), complete -o filenames marks candidates as file names so that directories get a slash and special characters are quoted, and complete -o default falls back to file completion when the function returns nothing.

zsh is the default login shell of macOS since Catalina and its completion system, compsys, is the most capable of the four. It is not active until the user’s ~/.zshrc runs autoload -U compinit && compinit, a fact that explains many reports of "my completion script does nothing".

The pieces are:

Source: Zsh Manual — Completion System

The script ends with _shelf "$@" because the same file can be used two ways: autoloaded from fpath as a function body, or run when sourced. Where to install it:

The exact system directories differ between distributions and zsh builds. The user can print the real list with print -l $fpath, and the tool’s installation guide should say which one it used.

fish does not need a separate completion framework to be switched on, and it loads completions lazily by file name: the first time the user completes shelf, fish looks for shelf.fish in the directories of $fish_complete_path. In the order documented, those are ~/.config/fish/completions (user), /etc/fish/completions (administrator), ~/.local/share/fish/vendor_completions.d and /usr/share/fish/vendor_completions.d (software vendors), the completions bundled with fish and ~/.cache/fish/generated_completions (generated from man pages). The documentation notes that the paths can vary with the build, and suggests ~/.config/fish/completions for the user’s own scripts. A package should install into a vendor directory.

The language is a series of complete commands, one per candidate set, each with a condition:

-c names the command, -s a short and -l a long option, -a the candidates (a space-separated string, or a command substitution evaluated at completion time), -n a condition that must succeed for the entry to apply, -d the description shown beside a candidate, -f turns file completion off for the command (or for one option) and -r says that the option requires a parameter. -x is the shorthand for exclusive: it combines -r and -f. __fish_seen_subcommand_from is documented helper function; __fish_use_subcommand is a function shipped with fish that is true while no subcommand has been typed, and it is used throughout fish’s own bundled completions, but the completion documentation page does not describe it.

Source: fish documentation — Writing your own completions

The fish completion is the one most often generated in a form that is eval-able: shelf completion fish | source in config.fish works, as well as saving to the completions directory. Because fish loads on demand, saving the file is better, since the directory costs nothing at shell start.

PowerShell completes its own cmdlets from metadata, so a native program such as shelf.exe gets only file names unless the user registers a native argument completer. Register-ArgumentCompleter -Native -CommandName shelf -ScriptBlock { ... } does that. For a native completer the script block receives three positional parameters: the word being completed ($wordToComplete), the abstract syntax tree of the current line ($commandAst) and the cursor position ($cursorPosition). It writes candidates to the pipeline, either plain strings or System.Management.Automation.CompletionResult objects, which carry the text to insert, the text to show in the list, a result type and a tooltip. The documentation specifies that the script block has to unroll its values through the pipeline, because returning an array would be treated as one candidate. Calling Register-ArgumentCompleter with -CommandName and no -ParameterName behaves as -Native; for PowerShell commands, always give -ParameterName.

Source: Register-ArgumentCompleter

The registration only lasts for the session, so it is dot-sourced from the user’s profile (the file that the $PROFILE variable names). The same script works on Windows PowerShell 5.1 and on PowerShell 7 on every platform. Two Windows-specific obstacles are worth documenting in the install instructions: a profile does not run if the execution policy forbids scripts, and Cobra’s and clap’s guides both tell Windows users to set the policy to RemoteSigned for the current user (Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser); and $PROFILE differs between Windows PowerShell and PowerShell 7 (they are different files), so a script installed for one shell does not complete in the other.

PowerShell 7.6 (since 7.6-preview.5) added -NativeFallback, a single cover-all completer for native commands that have no completer of their own, which is how a module can offer Unix-style completion for many tools at once; a tool author does not use it.

cmd.exe has no programmable completion. Tab completes file names only, so a Windows tool’s completion story is PowerShell, plus bash or zsh for users of Git for Windows or WSL.

Nushell separates the completer from the command definition. A completer is an ordinary Nushell command that returns a list of values, or a list of records with a value and optional description, or a record with completions and options (filter algorithm, case sensitivity, sort order). It is attached to an argument with the shape@completer annotation, and a non-Nushell program is described to Nushell with an extern definition, which lists its subcommands, flags and argument types so that Nushell can complete them and type-check them:

Source: Nushell — Custom Completions

Tools that cannot be described one by one use the external completer, a closure in $env.config.completions.external.completer that Nushell calls for external commands which have no extern or custom completer. The Nushell documentation shows it delegating to Carapace (a separate project that offers completions for many tools in many shells), and commandline complete lets a script author test a completer without pressing Tab. A tool that wants to support Nushell either ships a .nu file like the one above, or lets the user generate it from a completion subcommand; which frameworks emit Nushell is a matter for each framework’s documentation.

On Windows, the most robust distribution of PowerShell completion is a PowerShell module, or a completion subcommand that prints the script (the cross-platform approach in the AI section), because neither winget nor MSI has a standard place for a completion file. Linux packages (deb, rpm) and Homebrew formulas do have such places and should install the static scripts there; the packaging pages of this section describe how to place files in those packages (Native Packages & Installers).

Static scripts cannot complete shelf loan <id> with the real ids. Three families of dynamic completion exist, and they share one idea: the shell script is a thin trampoline that calls the tool in a special mode.

The same line reaches a program as different argv arrays depending on the shell, and a completion function must return candidates in the form that the shell then parses correctly. The rules that matter for a CLI author:

To see the difference for a title that contains an apostrophe, a double quote and a dollar sign, bash prints the same text in two forms:

The second line is what a bash completion function must hand back for the shell to read it as one word. This is the problem that compopt -o filenames, _filedir and zsh’s _describe solve for the author. A candidate list produced by a tool should be raw and unquoted, and quoting should be done by the shell script or framework.

Two consequences for the tool itself. A CLI that accepts a free-text argument (shelf add "The Hobbit") receives exactly one word in a POSIX shell, but receives the quotes as part of the string in a cmd.exe-launched legacy program if the program parses GetCommandLine itself instead of using the runtime’s argv. And anything a script builds from untrusted text, such as a title read from a file, must pass through the target shell’s quoting rules, not through a hand-made replace of ".

Source: POSIX Shell Command Language — Quoting and about_Parsing

A child process cannot change the working directory, the environment variables or the aliases of the process that started it. A tool that is meant to do exactly that (z, direnv, nvm, shelf cd fiction) is therefore implemented as two pieces: the real program, which only prints, and a shell function with the same name, defined in the user’s interactive shell, which runs the program and acts on its output. The user installs the function by evaluating the program’s own output:

The init subcommand is a normal program that prints a shell-specific snippet and exits with status 2 for a shell it does not know. The snippet for bash and zsh is the same:

command shelf bypasses the function and runs the executable, builtin cd is the real cd and -- stops an id that begins with a dash from being read as an option. The test run on the writing machine, with a stub path subcommand, printed the new directory in both bash and zsh after shelf cd fiction:

The same idea in fish and PowerShell uses their own syntax (function shelf ... end and function shelf { ... }), and Set-Location replaces cd in PowerShell. For a prompt that shows the state of the tool, the snippet also registers a hook (PROMPT_COMMAND in bash, precmd in zsh, fish_prompt in fish); the hook runs on every prompt and must therefore be much faster than a Tab completion. No Windows cmd.exe equivalent exists except doskey macros and batch files that are run with call.

Source: GNU Bash — Shell Builtin Commands (command, builtin, eval) and POSIX Shell Command Language

The design caveats come from eval: the output of shelf init is executed with the user’s authority in every new shell, so the program must be a trusted binary found through a trusted PATH, the snippet must not be built from user data, and it must be quick, since it adds to the start-up time of every interactive shell.

Scheduled and supervised jobs do not read ~/.bashrc, do not load the completion scripts, do not have the user’s PATH, PWD, aliases or terminal, and often run as a different user. Every behaviour of the tool that depended on the interactive shell then has to be spelled out.

A tool that is going to be scheduled should, independently of the scheduler: take all of its configuration from flags or an explicit config file path, write nothing to a terminal it does not have (see TTY, Colour & Interactivity), exit with a meaningful status (Streams & Exit Codes), and not prompt. A daemon that needs the full service lifecycle, not a periodic run, is covered in Long-Running Processes & Services.

A systemd timer for a nightly shelf sync, as a pair of units:

Source: systemd.timer and systemd.exec

The equivalent crontab entry, with its environment declared and the percent sign escaped:

Source: crontab(5)

And a launchd agent for the same job on macOS:

Source: Creating Launch Daemons and Agents (an archived document, but the only Apple page that describes these keys; RunAtLoad and EnvironmentVariables are described in the launchd.plist(5) man page)

On Windows, schtasks.exe or the Register-ScheduledTask cmdlet creates the task from a command line or a script; neither was run here. The Task Scheduler overview documents the triggers and the two API generations, but it does not describe the environment of a running task, so the table above only states what that page supports.

Source: Task Scheduler for developers