How does Pants work?

---

The Pants Engine

Pants 2.x is built around the "v2" engine, which is a complete, ground-up redesign of the entire system, based on lessons learned from 10 years of development on the previous, "v1", technology.

The Pants engine is written in Rust, for performance. The build rules that it uses are written in typed Python 3, for familiarity and simplicity. The engine is designed so that fine-grained invalidation, concurrency, hermeticity, caching, and remote execution happen naturally, without rule authors needing to think about it.

What are the benefits?

Concurrency

The engine can take full advantage of all the cores on your machine because relevant portions are implemented in Rust atop the Tokio framework.

Pants running multiple linters in parallel.

This means, for example, that you can run all of your linters at the same time, and fully utilize your cores to run tests in parallel.

Caching

The engine caches processes precisely based on their inputs, and sandboxes execution to minimize side-effects and to make builds consistent and repeatable.

We run both tests, then add a syntax error to one test and rerun; the unmodified test uses the cache and is isolated from the syntax error.

Fine-grained invalidation

Work is broken down into many small units and kept warm in a daemon so that as little work as possible needs to be re-done when files change.

Hermetic execution

Pants sandboxes all processes that it executes, ensuring that cache keys are always accurate, and builds are always correct.

Dependency inference

Pants analyzes your code's import statements to determine files' dependencies automatically. Dependency information is required for precise change detection and cache invalidation, but inference means that you don't need to declare dependencies manually (and hermetic execution guarantees that they are always accurate)!

Before (Pants v1, other build tools like Bazel):

BUILD

python_library(
  dependencies=[
    'src/python/pants/backend/python/rules',
    'src/python/pants/backend/python/subsystems',
    'src/python/pants/backend/awslambda/common',
    'src/python/pants/build_graph',
    'src/python/pants/core/util_rules',
    'src/python/pants/engine:addresses',
    'src/python/pants/engine:fs',
    'src/python/pants/engine:process',
    'src/python/pants/engine:rules',
    'src/python/pants/engine:selectors',
    'src/python/pants/engine:target',
    'src/python/pants/engine:unions',
    'src/python/pants/python',
  ],
)

python_tests(
  name='tests',
  dependencies=[
    ...
  ],
)

After:

BUILD

python_library()

python_tests(name='tests')

Remote Execution

Remote caching enables your coworkers and your CI to reuse the results of commands you already ran. Pants dependency inference gives you a remote execution plan for free. The engine can delegate work to a remote build cluster so that you are no longer limited by the number of cores on your machine. If you have enough remote workers, you can run your entire test suite in total parallelism.

A powerful plugin system

With the Pants plugin API, your custom rules will run with the same concurrency, caching, and remoting semantics as the core rules.

Some example plugins that users have written:

• Cython support
• Building a Docker image, including packages built via ./pants package
• Custom setup.py logic to compute the version dynamically
• Jupyter support