Version: 2.11 (deprecated)

Docker overview

How to build Docker images containing artifacts built by Pants

Docker images typically bundle build artifacts, such as PEX files, wheels, loose files, and so on, with other runtime requirements, such as a Python interpreter.

Pants makes it easy to embed the artifacts Pants builds into your Docker images, for easy deployment.

Enabling the Docker backend

To use Pants's Docker support you must enable the appropriate backend:

pants.toml
backend_packages = [
  ...
  "pants.backend.docker",
  ...
]

Adding `docker_image` targets

A Docker image is built from a recipe specified by a Dockerfile. When you build Docker images with Pants, instead of running docker on the Dockerfile directly, you let Pants do that for you.

Pants uses docker_image targets to indicate which Dockerfiles you want Pants to know about, and to add any necessary metadata.

You can generate initial BUILD files for your Docker images, using tailor:

❯ ./pants tailor
Created src/docker/app1/BUILD:
  - Add docker_image target docker
Created src/docker/app2/BUILD:
  - Add docker_image target docker

Or you can add them manually, such as:

src/docker/app1/BUILD
docker_image(name="docker")

Alternatively you may provide the Docker build instructions inline in your BUILD file as instructions on your docker_image if you don't want to create a Dockerfile.

src/docker/app1/BUILD
docker_image(
  name="docker",
  instructions=[
    "FROM python:3.8",
    "RUN ..",
  ]
)

The docker_image instructions field

Each docker_image uses a Dockerfile referred to by the source field, unless you have provided a value to the instructions field.

When using the instructions field, make sure that the default value for source does not match a file, or there will be a conflict about which information to use.

Adding dependencies to your `docker_image` targets

A Dockerfile executes in a context - a set of files that the commands in the Dockerfile can reference, e.g., by copying them into the image).

When you run docker directly, the context is usually a directory within your repo. That directory must contain the Dockerfile (typically at the root of the context) and any files that the build requires. If those files are themselves the product of a build step, or if they are sources from elsewhere in the repo, then you have to copy them into the context.

Pants, however, takes care of assembling the context for you. It does so using the dependencies of the docker_image target.

A docker_image can depend on loose files belonging to file / files targets, and on artifacts packaged from a variety of targets, such as pex_binary , python_distribution, archive, or any other target that can be built via the package goal.

The context is assembled as follows:

The sources of file / files targets are assembled at their relative path from the repo root.
The artifacts of any packageable targets are built, as if by running ./pants package, and placed in the context using the artifact's output_path field.
- The output_path defaults to the scheme path.to.directory/tgt_name.ext, e.g. src.python.helloworld/bin.pex.

Dependency inference for `pex_binary`

When you COPY PEX binaries into your image, the dependency on the pex_binary target will be inferred, so you don't have to add that explicitly to the list of dependencies on your docker_image target.

For example, the pex_binary target src/python/helloworld/bin.pex has the default output_path of src.python.helloworld/bin.pex. So, Pants can infer a dependecy based on the line COPY src.python.helloworld/bin.pex /bin/helloworld.

Building a Docker image

You build Docker images using the package goal:

❯ ./pants package path/to/Dockerfile

Build arguments

To provide values to any build ARGs in the Dockerfile, you can list them in the [docker].build_args option, which will apply for all images. You can also list any image-specific build args in the field extra_build_args for the docker_image target.

The build args use the same syntax as the docker build --build-arg command line option: VARNAME=VALUE, where the value is optional, and if left out, the value is taken from the environment instead.

pants.toml
example/BUILD
example/Dockerfile

[docker]
build_args = [
  "VAR1=value1",
  "VAR2"
]

docker_image(
  name="docker",
  extra_build_args=["VAR1=my_value", "VAR3"]
)

FROM python:3.8
ARG VAR1
ARG VAR2
ARG VAR3=default
...

Target build stage

When your Dockerfile is a multi stage build file, you may specify which stage to build with the --docker-build-target-stage for all images, or provide a per image setting with the docker_image field target_stage.

FROM python:3.8 AS base
RUN <install required tools>

FROM base AS img
COPY files /

❯ ./pants package --docker-build-target-stage=base Dockerfile

Build time secrets

Secrets are supported for docker_image targets with the secrets field. The defined secrets may then be mounted in the Dockerfile as usual.

BUILD

docker_image(
  secrets={
    "mysecret": "mysecret.txt",
  }
)

FROM python:3.8

# shows secret from default secret location:
RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret

# shows secret from custom secret location:
RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar

mysecret.txt

very-secret-value

Secret file path

Secrets should not be checked into version control. Use absolute paths to reference a file that is not in the project source tree. However, to keep the BUILD file as hermetic as possible, the files may be placed within the project source tree at build time for instance, and referenced with a path relative to the project root by default, or relative to the directory of the BUILD file when prefixed with ./.

See the example for the secrets field.

Build Docker image example

This example copies both a file and pex_binary:

src/docker/hw/BUILD
src/docker/hw/Dockerfile
src/docker/hw/msg.txt
src/py/hw/BUILD
src/py/hw/main.py

file(name="msg", source="msg.txt")

docker_image(
    name="docker",
    dependencies=[":msg", "src/python/hw:bin"],
)

FROM python:3.8
ENTRYPOINT ["/bin/helloworld"]
COPY src/docker/hw/msg.txt /var/msg
COPY src.python.hw/bin.pex /bin/helloworld

Hello, Docker!

python_sources(name="lib")

pex_binary(name="bin", entry_point="main.py")

import os

msg = "Hello"
if os.path.exists("/var/msg"):
    with open("/var/msg") as fp:
        msg = fp.read().strip()

print(msg)

❯ ./pants package src/docker/hw/Dockerfile
[...]
18:07:29.66 [INFO] Completed: Building src.python.hw/bin.pex
18:07:31.83 [INFO] Completed: Building docker image helloworld:latest
18:07:31.83 [INFO] Built docker image: helloworld:latest

Running a Docker image

You can ask Pants to run a Docker image on your local system with the run goal:

❯ ./pants run src/docker/hw/Dockerfile
Hello, Docker!

Any arguments for the Docker container may be provided as pass through args to the run goal, as usual. That is, use either the --args option or after all other arguments after a separating double-dash:

❯ ./pants run src/docker/hw/Dockerfile -- arguments for the container
Hello, Docker!

To provide any command line arguments to the docker run command, you may use the --docker-run-args option:

❯ ./pants run --docker-run-args="-p 8080 --name demo" src/docker/hw/Dockerfile

As with all configuration options, this is not limited to the command line, but may be configured in a Pants rc file (such as pants.toml) in the [docker].run_args section or as an environment variable, PANTS_DOCKER_RUN_ARGS as well.

Publishing images

Pants can push your images to registries using ./pants publish:

❯ ./pants publish src/docker/hw:helloworld
# Will build the image and push it to all registries, with all tags.

See here for how to set up registries.

Docker configuration

To configure the Docker binary, set [docker].env_vars in your pants.toml configuration file. You use that key to list environment variables such as DOCKER_CONTEXT or DOCKER_HOST, that will be set in the environment of the docker binary when Pants runs it. Each listed value can be of the form NAME=value, or just NAME, in which case the value will be inherited from the Pants process's own environment.

pants.toml
[docker]
env_vars = [
  "DOCKER_CONTEXT=pants_context",
  "DOCKER_HOST"
]

Docker environment variables

See Docker documentation for the authoritative table of environment variables for the Docker CLI.

Docker authentication

To authenticate, you usually will need to:

Set up a Docker config file, e.g. ~/.docker/config.json.
Tell Pants about the config file by setting [docker].env_vars.
Tell Pants about any tools needed for authentication to work by setting [docker].tools.

For example, a config file using the GCloud helper might look like this:

{
	"credHelpers": {
		"europe-north1-docker.pkg.dev": "gcloud"
	}
}

Then, tell Pants to use this config by setting [docker].env_vars = ["DOCKER_CONFIG=%(homedir)s/.docker"] in pants.toml, for example.

Most authentication mechanisms will also require tools exposed on the $PATH to work. Teach Pants about those by setting the names of the tools in [docker].tools, and ensuring that they show up on your $PATH. For example, GCloud authentication requires dirname, readlink and python3.

pants.toml
# Example GCloud authentication.

[docker]
env_vars = ["DOCKER_CONFIG=%(homedir)s/.docker"]
tools = [
  "docker-credential-gcloud",
  "dirname",
  "readlink",
  "python3",
  # These may be necessary if using Pyenv-installed Python.
  "cut",
  "sed",
  "bash",
]

You may need to set additional environment variables with [docker].env_vars.

How to troubleshoot authentication

It can be tricky to figure out what environment variables and tools are missing, as the output often has indirection.

It can help to simulate a hermetic environment by using env -i. With credential helpers, it also helps to directly invoke the helper without Docker and Pants. For example, you can symlink the tools you think you need into a directory like /some/isolated/directory, then run the below:

❯ echo europe-north1-docker.pkg.dev | env -i PATH=/some/isolated/directory docker-credential-gcloud get
{
  "Secret": "ya29.A0ARrdaM-...-ZhScVscwTVtQ",
  "Username": "_dcgcloud_token"
}

Linting Dockerfiles with Hadolint

Pants can run Hadolint on your Dockerfiles to check for errors and mistakes:

❯ ./pants lint src/docker/hw/Dockerfile

This must first be enabled by activating the Hadolint backend:

pants.toml
[GLOBAL]
backend_packages = ["pants.backend.docker.lint.hadolint"]

Enabling the Docker backend​

Adding docker_image targets​

Adding dependencies to your docker_image targets​

Dependency inference for pex_binary​

Building a Docker image​

Build arguments​

Target build stage​

Build time secrets​

Build Docker image example​

Running a Docker image​

Publishing images​

Docker configuration​

Docker authentication​

Linting Dockerfiles with Hadolint​