protobuf grpc relative import path discrepancy in python

Question

I emit the proto compilation into a generated_code/ directory(say communicator_pb2.py and communicator_pb2_grpc.py). Now, the grpc output imports the python code like so import communicator_pb2 as communicator__pb2; without having to modify the import path to import generated_code.communicator_pb2 as communicator__pb2 or append generated_code to sys.path, what is the most legal way to import if I were to import like:

import generate_code.commuinicator_pb2 as communicator_pb2
import generate_code.commuinicator_pb2_grpc as communicator_pb2_grpc

I'm not sure I understand the problem. If the generated_code directory is on your sys.path either by running generated_code/.., by setting PYTHONPATH, or via direct manipulation, can't you just use from generated_code import communicator_pb2 from your application code? — Richard Belleville
– Richard Belleville, Commented Jul 9, 2020 at 19:36
@RichardBelleville: the problem is communicator_pb2_grpc.py imports communicator_pb2.py like import communicator_pb2 as communicator__pb2 . I know that as long as my PYTHONPATH consists of generated_code/ we are safe here. I was wondering if there was a way to hide this in __init__.py or something slightly more elegant. Sorry, I lack some fundamentals here. — Shashank Hegde
– Shashank Hegde, Commented Jul 9, 2020 at 23:20
In your __init__.py, you might try adding sys.path.insert(0, PATH_TO_PROTOBUF_DIR). This will ensure that the import command you mentioned above will be able to properly resolve the communicator_pb2 module. — Richard Belleville
– Richard Belleville, Commented Jul 22, 2020 at 18:11

CaptainNabla · Accepted Answer · 2023-10-25 06:18:07Z

3

I've come up with an elegant solution (probably quite similar to what @Godric was telling in his answer). Suppose you have the following directory structure

service
  ├── main.py
  └── generated_code/
       ├── __init__.py       
       ├── communicator.proto
       ├── communicator_pb2.py
       └── communicator_pb2_grpc.py

If you compile it the usual way, using grpc_tools (see here)

python -m grpc_tools.protoc --proto_path=service/generated_code --python_out=service/generated_code --grpc_python_out=service/generated_code

you would get the following import in generated_code/communicator_pb2_grpc.py

import communicator_pb2 as communicator__pb2

However, when you compile the files with

python -m grpc_tools.protoc --proto_path=. --python_out=. --grpc_python_out=. generated_code/communicator.proto

while sitting in service/ you obtain the desired result

import generated_code.communicator_pb2 as communicator_pb2

edited Oct 25, 2023 at 6:18

answered Aug 21, 2023 at 14:33

CaptainNabla

1,1761 gold badge8 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

stranger Over a year ago

I have a project in which my primary .proto file (generated_code/iot-broker.proto in your example) imports a second file. When I invoke python -m grpc_tools.protoc per your recommendation here the import is not found. Do you know of a solution for this?

CaptainNabla Over a year ago

Sorry I haven't looked into cross references to other proto files

Godric · Accepted Answer · 2021-09-29 14:48:05Z

2

The command python -m grpc_tools.protoc ... myfile.proto chooses python package based on relative path to myfile.proto. So to use generated_code as package, you can:

cp myfile.proto generated_code/myfile.proto
python -m grpc_tools.protoc ... generated_code/myfile.proto

Or you can automatically patch the generated file to use relative import:

sed -i 's/^import .*_pb2 as/from . \0/' generated_code/commuinicator_pb2_grpc.py

answered Sep 29, 2021 at 14:48

Godric

212 bronze badges

1 Comment

stranger Over a year ago

Do you know how to get the patch-free solution to work when myfile.proto imports another .proto file? The relative path solution is breaking down for me in that case

Collectives™ on Stack Overflow

protobuf grpc relative import path discrepancy in python

2 Answers 2

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related