Skip to main content
Version: 2.5 (deprecated)

Overview

Pants's support for Shellcheck, shfmt, and shUnit2.

Pants integrates with these tools to empower you to follow best practices with your Shell scripts:

  • Shellcheck: lint for common Shell mistakes.
  • shfmt: autoformat Shell code so that you can instead focus on the logic.
  • shUnit2: write light-weight unit tests for your Shell code.

Pants installs these tools deterministically and integrates them into the workflows you already use: ./pants fmt, ./pants lint, and ./pants test.

Initial setup: add shell_library targets​

Pants uses shell_library targets to know which Shell files you want Pants to know about and to add additional metadata. It also uses shunit2_tests targets for test support.

First, activate the Shell backend in your pants.toml:

pants.toml
backend_packages = [
  "pants.backend.shell",
]

Then, run ./pants tailor on a clean branch to autogenerate BUILD files:

$ ./pants tailor
Created scripts/BUILD:
  - Added shell_library target scripts
Created scripts/subdir/BUILD:
  - Added shell_library target scripts/subdir

You can also manually add targets, which is necessary if you have any scripts that don't end in .sh:

shell_library(name="scripts", sources=["script_without_an_extension", "*.sh"])
Shell dependency inference

Pants will infer dependencies by looking for imports like source script.sh and . script.sh. You can check that the correct dependencies are inferred by running ./pants dependencies path/to/script.sh and ./pants dependencies --transitive path/to/script.sh.

Normally, Pants will not understand dynamic sources, e.g. using variable expansion. However, Pants uses Shellcheck for parsing, so you can use Shellcheck's syntax to give a hint to Pants:

another_script="dir/some_script.sh"

# Normally Pants couldn't infer this, but we can give a hint like this:
# shellcheck source=dir/some_script.sh
source "${another_script}"

Alternatively, you can explicitly add dependencies in the relevant BUILD file.

shell_library(dependencies=["path/to:shell_library"])

shfmt autoformatter​

To activate, add this to your pants.toml:

pants.toml
[GLOBAL]
backend_packages = [
  "pants.backend.shell",
  "pants.backend.shell.lint.shfmt",
]

Make sure that you also have set up shell_library and/or shunit2_tests targets so that Pants knows to operate on those files.

Now you can run ./pants fmt and ./pants lint:

$ ./pants lint scripts/my_script.sh
13:05:56.34 [WARN] Completed: lint - shfmt failed (exit code 1).
--- scripts/my_script.sh.orig
+++ scripts/my_script.sh
@@ -9,7 +9,7 @@

 set -eo pipefail

-HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" && \
+HERE=$(cd "$(dirname "${BASH_SOURCE[0]}")" &&
   pwd)

𐄂 shfmt failed.

Pants will automatically include any relevant .editorconfig files in the run. You can also pass command line arguments with --shfmt-args='-ci -sr' or permanently set them in pants.toml:

[shfmt]
args = ["-i 2", "-ci", "-sr"]

Temporarily disable shfmt with --shfmt-skip:

./pants --shfmt-skip fmt ::
Benefit of Pants: shfmt runs in parallel with Python formatters

Normally, Pants runs formatters sequentially so that it can pipe the results of one formatter into the next. However, Pants will run shfmt in parallel to formatters for other languages, like Python, because shfmt does not operate on those languages.

You can see this concurrency through Pants's dynamic UI.

Shellcheck linter​

To activate, add this to your pants.toml:

pants.toml
[GLOBAL]
backend_packages = [
  "pants.backend.shell",
  "pants.backend.shell.lint.shellcheck",
]

Make sure that you also have set up shell_library and/or shunit2_tests targets so that Pants knows to operate on those files.

Now you can run ./pants lint:

$ ./pants lint scripts/my_script.sh
13:09:10.49 [WARN] Completed: lint - Shellcheck failed (exit code 1).

In scripts/my_script.sh line 12:
HERE=$(cd $(dirname ${BASH_SOURCE[0]}) && pwd)
          ^--------------------------^ SC2046: Quote this to prevent word splitting.
                    ^---------------^ SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
...

𐄂 Shellcheck failed.

Pants will automatically include any relevant .shellcheckrc and shellcheckrc files in the run. You can also pass command line arguments with --shellcheck-args='-x -W 3' or permanently set them in pants.toml:

[shellcheck]
args = ["--external-sources", "--wiki-link-count=3"]

Temporarily disable Shellcheck with --shellcheck-skip:

./pants --shellcheck-skip lint ::
Benefit of Pants: Shellcheck runs in parallel with other linters

Pants will attempt to run all activated linters and formatters at the same time for improved performance, including Python linters. You can see this through Pants's dynamic UI.

shUnit2 test runner​

shUnit2 allows you to write lightweight unit tests for your Shell code.

To use shunit2 with Pants:

  1. Create a test file like tests.sh, test_foo.sh, or foo_test.sh.
  2. Create a shunit2_tests target in the directory's BUILD file.
  3. Specify which shell to run your tests with, either by setting a shebang directly in the test file or by setting the field shell on the shunit2_tests target.
    • See here for all supported shells.
#!/usr/bin/env bash

testEquality() {
  assertEquals 1 1
}

You can then run your tests like this:

# Run all tests in the repository.
./pants test ::

# Run all the tests in this target.
./pants test scripts:tests

# Run just the tests in this file.
./pants test scripts/tests.sh

Pants will download the ./shunit2 script and will add source ./shunit2 with the correct relpath for you.

You can import your production code by using source. Make sure the code belongs to a shell_library target. Pants's dependency inference will add the relevant dependencies, which you can confirm by running ./pants dependencies scripts/tests.sh. You can also manually add to the dependencies field of your shunit2_tests target.

#!/usr/bin/bash

source scripts/lib.sh

testAdd() {
    assertEquals $(add_one 4) 5
}
Running your tests with multiple shells

Pants allows you to run the same tests against multiple shells, e.g. Bash and Zsh, to ensure your code works with each shell.

To test multiple shells, create a distinct shunit2_tests target for each shell, like this:

shunit2_tests(name="tests_bash", shell="bash")
shunit2_tests(name="tests_zsh", shell="zsh")

Then, use ./pants test:

# Run tests with both shells.
./pants test scripts/tests.sh

# Run tests with only Zsh.
./pants test scripts/tests.sh:tests_zsh

Controlling output​

By default, Pants only shows output for failed tests. You can change this by setting --test-output to one of all, failed, or never, e.g. ./pants test --output=all ::.

You can permanently set the output format in your pants.toml like this:

pants.toml
[test]
output = "all"

Force reruns with --force​

To force your tests to run again, rather than reading from the cache, run ./pants test --force path/to/test.sh.

Setting environment variables​

Test runs are hermetic, meaning that they are stripped of the parent ./pants process's environment variables. This is important for reproducibility, and it also increases cache hits.

To add any arbitrary environment variable back to the process, use the option extra_env_vars in the [test] options scope. You can hardcode a value for the option, or leave off a value to "allowlist" it and read from the parent ./pants process's environment.

pants.toml
[test]
extra_env_vars = ["VAR1", "VAR2=hardcoded_value"]

Use [bash-setup].executable_search_paths to change the $PATH env var used during test runs. You can use the special string "<PATH>" to read the value from the parent ./pants process's environment.

pants.toml
[bash-setup]
executable_search_paths = ["/usr/bin", "<PATH>"]

Timeouts​

Pants can cancel tests that take too long, which is useful to prevent tests from hanging indefinitely.

To add a timeout, set the timeout field to an integer value of seconds, like this:

BUILD
shunit2_tests(name="tests", timeout=120)

This timeout will apply to each file belonging to the shunit2_tests target, meaning that test_f1.sh will have a timeout of 120 seconds and test_f2.sh will have a timeout of 120 seconds. If you want finer-grained timeouts, create a dedicated shunit2_tests target for each file:

BUILD
shunit2_tests(
    name="test_f1",
    sources=["test_f1.sh"],
    timeout=20,
)

shunit2_tests(
    name="test_f2",
    sources=["test_f2.sh"],
    timeout=35,
)

Unlike with Python, you cannot yet set a default or maximum timeout value, nor temporarily disable all timeouts. Please let us know if you would like this feature.

Testing your packaging pipeline​

You can include the result of ./pants package in your test through the runtime_package_dependencies field. Pants will run the equivalent of ./pants package beforehand and copy the built artifact into the test's chroot, allowing you to test things like that the artifact has the correct files present and that it's executable.

This allows you to test your packaging pipeline by simply running ./pants test ::, without needing custom integration test scripts.

To depend on a built package, use the runtime_package_dependencies field on the shunit2_tests target, which is a list of addresses to targets that can be built with ./pants package, such as pex_binary, python_awslambda, and archive targets. Pants will build the package before running your test, and insert the file into the test's chroot. It will use the same name it would normally use with ./pants package, except without the dist/ prefix.

For example:

python_library()
pex_binary(name="bin", entry_point="say_hello.py")

shunit2_tests(
    name="tests",
    runtime_package_dependencies=[":bin"],
)