Developing Rust

Hacking on the Pants engine in Rust.

---

We welcome contributions to Rust! We use Rust to implement the Pants engine in a performant, safe, and ergonomic way.

Still learning Rust? Ask to get added to reviews

We'd be happy to ping you on Rust changes we make for you to see how Rust is used in the wild. Please message us on the #engine channel in Slack to let us know your interest.

Recommendation: share your plan first

Because changes to Rust deeply impact how Pants runs, it is especially helpful to share any plans to work on Rust before making changes. Please message us on Slack in the #engine channel or open a GitHub issue.

Code organization

The code for the top-level Pants Rust crate lives in src/rust/engine. The top-level Cargo.toml file at src/rust/engine/Cargo.toml defines a cargo workspace containing a number of other subcrates, which live in subdirectories of src/rust/engine. Defining multiple subcrates in this way allows changes affecting one subcrate to avoid affecting other subcrates and triggering more recompilation than is necessary.

Several of the particularly important subcrates are:

• graph: the core of Pants's rule graph implementation.
• ui: the dynamic UI.
• sharded_lmdb: custom wrappers around the crates.io lmdb crate, which provides bindings to lmdb.
• fs: manipulating the filesystem.
• process_execution: running local and remote processes.

Rust <-> Python interaction

Pants is best conceptualized as a Python program that makes frequent foreign function interface (FFI) calls into Rust code.

The top-level engine Rust crate gets compiled into a library named native_engine.so, which Python code knows how to interact with. We use the Rust cpython crate to manage foreign function interaction.

The C FFI functions that Rust code exposes as a public interface live in src/rust/engine/src/externs/interface.rs. On the Python side, the native module at src/python/pants/engine/internals/native.py contains wrappers around invocations of Rust functions and other FFI boilerplate.

Rust can also invoke Python functions and object constructors thanks to the cpython crate. Most of our helper code lives in src/rust/engine/src/externs/mod.rs.

We are planning to port additional functionality from Python to Rust, generally for performance reasons.

Common commands

Compile

Run ./pants or build-support/bin/native/cargo check --manifest-path src/rust/engine/Cargo.toml.

Set MODE=debug when iterating

As described in Setting up Pants, we default to compiling Rust in release mode, rather than debug mode.

When working on Rust, you typically should set the environment variable MODE=debug for substantially faster compiles.

Run tests

To run tests for all crates, run:

build-support/bin/native/cargo test --manifest-path src/rust/engine/Cargo.toml

To run for a specific crate, such as the fs create, run:

build-support/bin/native/cargo test --manifest-path src/rust/engine/fs/Cargo.toml

To run for a specific test, use Cargo's filtering mechanism, e.g.:

build-support/bin/native/cargo test --manifest-path src/rust/engine/Cargo.toml read_file_missing

Autoformat

build-support/bin/check_rust_formatting.sh -f

To run in lint mode, rather than format mode, do not pass the -f flag.

Run Clippy

build-support/bin/check_clippy.sh

Cargo commands

We have a wrapper script around cargo at build_support/bin/native/cargo. Use this for all Cargo operations in the Pants codebase.

IDE setup (optional)

IntelliJ

1. Bootstrap Pants's Rust toolchain by running ./pants.
2. Install the Rust plugin (see IntelliJ Preferences).
3. Set Rust's Toolchain location to build-support/bin/native.
4. Set the Standard library to build-support/bin/native/src.

The fs_util tool

fs_util is a utility that enables you to interact with Snapshots from the command line. You can use it to help debug issues with snapshotted files.

To build it, run this from the root of the repository:

$ cd src/rust/engine && ../../../build-support/bin/native/cargo build -p fs_util

That will produce src/rust/engine/target/debug/fs_util.

To inspect a particular snapshot, you'll need to tell fs_util where the storage is and the digest and length of the snapshot to inspect. You can use the --local-store-path flag for that.

For example, this command pretty prints the recursive file list of a directory through the directory subcommand.

$ src/rust/engine/target/debug/fs_util --local-store-path=${HOME}/.cache/pants/lmdb_store directory cat-proto --output-format=recursive-file-list <digesh> <len>

Pass the --help flag to see other ways of using fs_util, along with its subcommands. Each subcommand can be passed the --help flag.