Skip to main content
Version: 2.23 (prerelease)

AWS Lambda

Create a Lambda with Python code.

Pants can create a Lambda-compatible zip file or directory from your Python code, allowing you to develop your Lambda functions and layers in your repository instead of using the online Cloud9 editor.

FYI: how Pants does this

Under-the-hood, Pants uses the PEX project, to select the appropriate third-party requirements and first-party sources and lay them out in a zip file or directory, in the format recommended by AWS.

Step 1: Activate the Python AWS Lambda backend​

Add this to your pants.toml:

pants.toml
[GLOBAL]
backend_packages.add = [
  "pants.backend.awslambda.python",
  "pants.backend.python",
]

This adds the new python_aws_lambda_function target, which you can confirm by running pants help python_aws_lambda_function

Step 2: Define a python_aws_lambda_function target​

First, add your lambda function in a Python file like you would normally do with AWS Lambda. Specifically, create a function def my_handler_name(event, context) with the name you want.

Then, in your BUILD file, make sure that you have a python_source or python_sources target with the handler file included in the sources field. You can use pants tailor :: to automate this.

Add a python_aws_lambda_function target and define the handler fields. The handler has the form handler_file.py:handler_func, which Pants will convert into a well-formed entry point. Alternatively, you can set handler to the format path.to.module:handler_func.

For example:

# The default `sources` field will include our handler file.
python_sources(name="lib")

python_aws_lambda_function(
    name="lambda",
    # Pants will convert this to `project.lambda_example:example_handler`.
    handler="lambda_example.py:example_handler",
)

Pants will use dependency inference based on the handler field, which you can confirm by running pants dependencies path/to:lambda. You can also manually add to the dependencies field.

You can optionally set the output_path field to change the generated zip file's path.

Using layout

Use layout to determine whether to build a .zip file or a directory

Use resource instead of file

file / files targets will not be included in the built AWS Lambda artifacts because filesystem APIs like open() would not load them as expected. Instead, use the resource and resources target. See Assets and archives for further explanation.

Specifying a runtime explicitly​

When building an Lambda artifact, Pants and the underlying Pex tool need to know details about target runtime to be able to choose appropriate artifacts for third-party dependencies that have native code. These details can be inferred or provided in three ways, from highest precedence to lowest precedence:

  1. An explicit value for the complete_platforms field. The "complete platforms" are the underlying source of truth.

    BUILD
    file(name="lambda-platform", source="lambda-platform.json")

    python_aws_lambda_function(
        name="lambda",
        handler="lambda_example.py:example_handler",
        # Explicit complete platforms:
        complete_platforms=[":lambda-platform"],
    )

    You can generate the complete_platforms file for a specific Lambda runtime by running the below cloud function and outputting its file contents. For Lambda, a handler to generate the file might look like this:

    import subprocess
    import json

    def lambda_handler(event, context):
        subprocess.run("pip install --target=/tmp/pex pex", shell=True)
        result = subprocess.run(
            "PYTHONPATH=/tmp/pex /tmp/pex/bin/pex3 interpreter inspect --markers --tags --indent=2",
            shell=True, capture_output=True, text=True
        )
        return {
            "statusCode": 200,
            "body": result.stdout,
        }

    Deploy and invoke the function, then retrieve the output to use as your complete_platforms file.

  2. An explicit value for the runtime field and, optionally, the architecture field: Pants uses these to pick an appropriate "complete platforms" value, from options that Pants has pre-packaged. These are static exports from docker images provided by AWS, relying on the environment being relatively stable. (If Pants doesn't have an appropriate "complete platforms" default built-in, you will be prompted to use option 1 above.)

    BUILD
    python_aws_lambda_function(
        name="lambda",
        handler="lambda_example.py:example_handler",
        # Explicit runtime, `complete_platforms` taken from Pants' built-in defaults:
        runtime="python3.12",
        # Override the default x86_64 architecture:
        architecture="arm64",
    )

  3. Inferred from the relevant interpreter constraints: the interpreter constraints may unambiguously imply a value for the runtime and thus complete_platforms fields. For example, interpreter_constaints = ["==3.12.*"] implies runtime="python3.12". This only works with interpreter constraints that cover all patch versions of a given minor release series: >=3.11,<3.13 is too wide (it covers both 3.11 and 3.12), while ==3.12.0 is too specific (AWS's python3.12 runtime may not use that exact patch version). As with option 2, the architecture is x86_64 by default, but can changed using the architecture field.

    [python]
    interpreter_constraints = ["==3.12.*"]

This guide is written using the last option, with the default x86_64 architecture, but you can add runtime, complete_platforms and/or architecture to any examples using the python_aws_lambda_function or python_aws_lambda_layer targets.

Step 3: Run package​

Now run pants package on your python_aws_lambda_function target to create a zipped file.

For example:

$ pants package project/:lambda
Wrote dist/project/lambda.zip
  Handler: lambda_function.handler
Running from macOS and failing to build?

AWS Lambda functions must run on Linux, so Pants tells PEX and Pip to build for Linux when resolving your third party dependencies. This means that you can only use pre-built wheels (bdists). If your project requires any source distributions (sdists) that must be built locally, PEX and pip will fail to run.

If this happens, you must either change your dependencies to only use dependencies with pre-built wheels or find a Linux environment to run pants package.

"Encountering collisions" errors and failing to build?

If a build fails with an error like Encountered collisions populating ... from PEX at faas_repository.pex:, listing one or more files with different sha1 hashes, this likely means your dependencies package files in unexpected locations, outside their "scoped" directory (for instance, a package example-pkg typically only includes files within example_pkg/ and example_pkg-*.dist-info/ directories). When multiple dependencies do this, those files can have exactly matching file paths but different contents, and so it is impossible to create a Lambda artifact: which of the files should be installed and which should be ignored? Resolving this requires human intervention to understand whether any of those files are important, and hence PEX emits an error rather than making an (arbitrary) choice that may result in confusing and/or broken behaviour at runtime.

Most commonly this seems to happen with metadata like a README or LICENSE file, or test files (in a tests/ subdirectory), which are likely not important at runtime. In these cases, the collision can be worked around by adding a pex3_venv_create_extra_args=["--collisions-ok"] field to the python_aws_lambda_... targets.

A better solution is to work with the dependencies to stop them from packaging files outside their scoped directories.

Step 4: Upload to AWS​

You can use any of the various AWS methods to upload your zip file, such as the AWS console or the AWS CLI via aws lambda create-function and aws lambda update-function-code.

You can specify the AWS lambda handler as lambda_function.handler. This is a re-export of the function referred to by the handler field of the target.

Docker Integration​

To deploy a Python lambda function with container images, you can use Pants's Docker support.

For example:

FROM public.ecr.aws/lambda/python:3.8

RUN yum install unzip -y
COPY project/lambda.zip .
RUN unzip lambda.zip -d "${LAMBDA_TASK_ROOT}"
CMD ["lambda_function.handler"]

Then, use pants package project:my_image, for example. Pants will first build your AWS Lambda function, and then will build the Docker image and copy it into the AWS Lambda.

Building a Lambda Layer​

AWS Lambda layers allow including additional code in the execution environment of a Lambda function, without having to include that code in the function package. Using a layer can allow for including more code in a single function, sharing common dependencies across several functions, and may even give faster builds and deploys.

Pants uses the python_aws_lambda_layer target to build AWS Lambda layers. The contents of the layer must be specified in the dependencies field, and Pants will pull in all of the code that implies (transitively) as usual, including any exclusions via ! and !!. The include_sources and include_requirements fields provide additional control over the contents of the layer.

For example, one use of layers is splitting the deployment package for a Lambda function into:

  1. a function artifact with only the code in your repository (first-party sources)
  2. a layer artifact with the third-party requirements that the function imports

This split means making a change to first-party sources only requires rebuilding and re-deploying the function artifact. Since this artifact doesn't need to include all of the third-party requirements, rebuilding is likely to be much faster and the resulting package will be smaller. The layer will only need to be rebuilt and redeployed if the third-party dependencies change, like a version upgrade or an additional import.

python_sources(name="lib")

python_aws_lambda_function(
    name="function",
    handler="lambda_example.py:example_handler",
    # only include the sources, the boto3 requirement is packaged in `:layer`
    include_requirements=False,
)

python_aws_lambda_layer(
    name="layer",
    # specify the handler file, and pants will automatically find its transitive dependencies
    dependencies=["./lambda_example.py"],
    # only include the boto3 requirement, any sources are packaged in `:function`
    include_sources=False,
)

Run pants package project:layer project:function to produce two zip files:

  • dist/project/layer.zip: this must be published as a layer in AWS, such as through the console or using the CLI (aws lambda publish-layer-version).
  • dist/project/function.zip: as above, this can be uploaded to AWS in various ways and the handler can be set to lambda_function.handler. The function will need specify that it uses the layer created above.

Advanced: Using PEX directly​

In the rare case where you need access to PEX features, such as dynamic selection of dependencies, a PEX file created by pex_binary can be used as a Lambda function package directly. A PEX file is a carefully constructed zip file, and can be understood natively by AWS. Note: using pex_binary results in larger packages and slower cold starts and is likely to be less convenient than using python_aws_lambda_function.

The handler of a pex_binary is not re-exported at the fixed lambda_function.handler path, and the Lambda function handler must be configured as the __pex__ pseudo-package followed by the handler's normal module path (for instance, if the handler is called func in some/module/path.py within a source root, then use __pex__.some.module.path.func). The __pex__ pseudo-package ensures dependencies are initialized before running any of your code.

For example:

python_sources()

pex_binary(
    name="lambda",
    entry_point="lambda_example.py",
    # specify an appropriate platform for the targeted Lambda runtime:
    complete_platforms=["path/to:platform-json-target"],
)

Then, use pants package project:lambda, and upload the resulting project/lambda.pex to AWS. The handler will need to be configured in AWS as __pex__.lambda_example.example_handler (assuming project is a source root).

Migrating from Pants 2.16 and earlier​

Pants implemented a new way to package Lambda functions in 2.17, which became the only option in 2.19, resulting in smaller packages and faster cold starts. This involves some changes:

  • In Pants 2.16 and earlier, Pants used the Lambdex project. First, Pants would convert your code into a Pex file and then use Lambdex to adapt this to be better understood by AWS by adding a shim handler at the path lambdex_handler.handler. This shim handler first triggers the Pex initialization to choose and unzip dependencies, during the "INIT" phase.
  • In Pants 2.17, the use of Lambdex was deprecated, in favour of choosing the appropriate dependencies ahead of time, as described above, without needing to do this on each cold start. This results in a zip file laid out in the format recommended by AWS, and includes a re-export of the handler at the path lambda_function.handler.
  • In Pants 2.18, the new behaviour is now the default behaviour. Layers can now be built using Pants, and this addition includes renaming the python_awslambda target to python_aws_lambda_function.
  • In Pants 2.19 and later, the old Lambdex behaviour has been entirely removed.

If your code can be packaged without warnings using Pants 2.18, no change is required when upgrading to Pants 2.19 (except removing the [lambdex] section in pants.toml if that still remains). If not, follow its instructions to upgrade to Pants 2.18 fully first, and upgrade to Pants 2.19 after that.

If you encounter a bug with the new behaviour, please let us know. If you require advanced PEX features, switch to using pex_binary directly.