Protobuf and gRPC

How to generate Python from Protocol Buffers.

---

When you depend on a protobuf_library in a Python target (like a python_library), Pants will run the Protoc compiler to generate Python code that you can import and use like normal Python code.

Example repository

See the Python example repository for an example of using Protobuf to generate Python.

Step 1: Activate the Protobuf Python backend

Add this to your pants.toml:

pants.toml

[GLOBAL]
backend_packages.add = [
  "pants.backend.codegen.protobuf.python",
  "pants.backend.python",
]

This adds the new protobuf_library target, which you can confirm by running ./pants help protobuf_library.

Enable the MyPy Protobuf plugin

The MyPy Protobuf plugin generates .pyi type stubs. If you use MyPy through Pants's typecheck goal, this will ensure MyPy understands your generated code.

To activate, set mypy_plugin = true in the [python-protobuf] scope:

[python-protobuf]
mypy_plugin = true

MyPy will use the generated .pyi type stub file, rather than looking at the .py implementation file.

Want to use other protocols, like Thrift?

Please message us on Slack if you would like support for more protocols. We would be happy to either add support to the core Pants distribution or to help you to write a plugin.

Step 2: Set up the protobuf and/or grpcio runtime libraries

Generated Python files require the protobuf library for their imports to work properly. If you're using gRPC, you also need the grpcio library.

First, add protobuf—and grpcio, if relevant— to your requirements.txt (see Third-party dependencies).

requirements.txt

grpcio==1.32.0
protobuf>=3.12.1

Then, add the targets' addresses to the option runtime_dependencies in the [python-protobuf] scope. Pants will use this to automatically add the target(s) to the dependencies field for every protobuf_library() target you write.

pants.toml

[python-protobuf]
# Use the path to your 3rd-party requirements,
# e.g. `3rdparty/python:protobuf`.
runtime_dependencies = ["//:grpcio", "//:protobuf"]

Step 3: Define a protobuf_library target

Wherever you create your .proto files, add a protobuf_library.

src/proto/example/BUILD

# `sources` defaults to `['*.proto']`.
protobuf_library()

src/proto/example/f.proto

syntax = "proto3";

package example;

message Example {
  ...
}

Your protobuf_library can optionally depend on other protobuf_library targets through the dependencies field, if its .proto files need to import definitions from other.proto files.

If you want gRPC code generated, set grpc=True.

src/proto/example/BUILD

protobuf_library(grpc=True)

Step 4: Use the protobuf_library in your Python code

Now, you can add the protobuf_library to the dependencies field of your Python targets. Pants will generate the Python code automatically for you.

In your Python file, import the module with the name _pb2 at the end, e.g. protos/example.proto becomes proto.example_pb2.

If gRPC is activated, you can also import the module with _pb2_grpc at the end, e.g. proto.example_pb2_grpc.

src/python/example/BUILD

python_library(
    dependencies=[
        "src/proto/example",
    ],
)

src/python/example/app.py

from example.f_pb2 import Message

# See https://developers.google.com/protocol-buffers/docs/pythontutorial
# for how to use the generated code in your project.

Dependency inference does not work with Protobuf. You must explicitly declare all dependencies on protobuf_library targets.

You likely need to add empty __init__.py files

By default, Pants will generate the Python files in the same directory as the .proto file. To get Python imports working properly, you will likely need to add an empty __init__.py in the same location, and possibly in ancestor directories. You do not need to add a python_library() target; Pants will automatically include the file.

See the below section "Protobuf and source roots" for how to generate into a different directory. If you use this option, you will still likely need an empty __init__.py file in the destination directory.

Upcoming feature: export-codegen goal

Pants 2.1 adds an export-codegen rule to write the files to the dist/ dir.

If you are not yet able to upgrade to 2.1, for now, you can add the protobuf_library() as a dependency of a pex_binary(), then run ./pants package on the binary target, and finally inspect the built PEX by using unzip.

Upcoming feature: dependency inference for Protobuf

Pants 2.2 adds support for dependency inference of:

• Python imports of Protobuf files, including gRPC files.
• Protobuf dependencies on other Protobuf files.

Protobuf and source roots

By default, generated code goes into the same source root as the .proto file from which it was generated. For example, a file src/proto/example/f.proto will generate src/proto/example/f_pb2.py.

However this may not always be what you want. In particular, you may not want to have to add __init__py files under src/proto just so you can import Python code generated to that source root.

You can configure a different source root for generated code by setting the python_source_root field:

src/proto/example/BUILD

protobuf_library(
  python_source_root='src/python'
)

Now src/proto/example/f.proto will generate src/python/example/f_pb2.py, i.e., the generated files will share a source root with your other Python code.

Set packages relative to the source root

Remember that the package directive in your .proto file should be relative to the source root.

For example, if you have a file at src/proto/example/subdir/f.proto, you'd set its package to example.subdir; and in your Python code, from example.subdir import f_pb2.