Can the Python `protoc` compiler module from `grpc_tools` automatically include proto files from site-packages?

Question

In Python, the gRPC project proto files are shipped in smaller packages. For example, I want to use status.proto, which is the grpc-status package.

The protoc compiler is provided as a Python module by the grpc-tools package.

# my.proto
syntax = "proto3";
import "google/rpc/status.proto";

Installing everything in a virtualenv:

python -m venv v
source v/bin/activate
pip install grpc-status grpc-tools

The compiler module doesn't automatically find and use the proto files installed by the grpc-status package.

python -m grpc_tools.protoc -I . my.proto

Results in:

google/rpc/status.proto: File not found.

(This file is available at v/lib/python3.11/site-packages/google/rpc/status.proto and was installed by googleapis-common-protos, a dependency of grpcio-status.)

This surprises me, because the proto files are distributed in separate packages, and the protoc compiler is itself a Python module, and so the entire arrangement for "provided" proto files seems easy and designed to be configured within Python by grpc_tools itself. It seems I must be doing something wrong.

Do I really need to explicitly tell the compiler module about Python's site-packages?

SITE_PROTO_FILES=$( python -c 'import site; print(site.getsitepackages()[0])' )

# This compiles.
python -m grpc_tools.protoc -I $SITE_PROTO_FILES -I . my.proto

I have searched the gRPC documentation, Google Group GitHub Issue tracker, and read python -m grpc_tools.protoc --help, but have not found anything useful about this.

DazWilkin · Accepted Answer · 2024-08-11 15:52:47Z

Yes, you must explicitly tell protoc (python -m grpc_tools.protoc) about any proto paths except usually Google Well-Known Types (WKTs i.e. google/protobuf/...) as these are included by default.

Using protoc without a complete set of paths generates an error about the complexity in parsing proto paths and the necessity of including them.

It begs the question as to why the protobuf sources are included alongside the protoc-generated Python sources but this is good practice/hygiene; you have the source code that matches the compiled Python sources.

Because you have the Python sources (e.g. status_pb2.py and code_pb2.py), you generally wouldn't need to access the sources.

But, because your example, defines a new message that imports google/grpc/status.proto, you need to reference it per the above requirements (you must --proto_path everything except WKTs and protoc is unable to do this automatically).

Collectives™ on Stack Overflow

Can the Python `protoc` compiler module from `grpc_tools` automatically include proto files from site-packages?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related