Python gRPC: Cannot import <generated module> from proto

Question

I have a gRPC service, for which python grpc compiler generated two files in the proto directory:

service_pb2.py
service_pb2_grpc.py

In the service_pb2_grpc.py there is an import from proto import service_pb2 as proto_dot__service__pb2. This import fails because in my virtualenv I have another package called proto which seems to be a dependency of google cloud logging library. So import is attempted from site package, not from the proto directory. I don't want to run some script after compiler does its job, to rename this generated proto directory. Is there a way to tell the grpc compiler which name to use for proto containing folder?

Did you try relative imports instead of absolute? This is not the best way to solve it tho. The best way would be renaming your proto folder — sudden_appearance
– sudden_appearance, Commented Apr 14, 2022 at 9:55
Sure, it would be better to rename the folder. But, I will also have to rename the proto files, which scream at me "DO NOT EDIT". Thus I don't like this approach of manual renaming. — Pavlo Kovalov
– Pavlo Kovalov, Commented Apr 14, 2022 at 9:56
Then wrap it with one more package. For it to be package.proto.service_pb2 — sudden_appearance
– sudden_appearance, Commented Apr 14, 2022 at 9:58
The import inside generated files will still be from proto import ..., even if proto directory will be inside another package, right? — Pavlo Kovalov
– Pavlo Kovalov, Commented Apr 14, 2022 at 9:59

newbnoob · Accepted Answer · 2024-04-24 18:50:36Z

if anyone is dealing with this in 2024, the 'official workaround' is to append the sys.path with the path of the .proto file parent directory, allowing the import statements in service_pb2_grpc.py to be processed properly.

Run the protoc command and create an __init__.py file under the parent directory of .proto file.

Here is a script that can be used to recursively process all .proto files under a specified directory, generating the protobuf files and creating necessary __init__.py to resolve import issues.


import os
import subprocess


from client import PROJECT_ROOT

proto_files_dir = os.path.join(PROJECT_ROOT, 'client', 'api')
for dirpath, dirnames, filenames in os.walk(proto_files_dir):
    proto_files = [f for f in filenames if f.endswith('.proto')]
    if proto_files:
        # For each .proto file, run the protoc command to generate the Python files
        for proto_file in proto_files:
            print(f"Processing .proto file '{proto_file}'...", end=' ')
            proto_file_path = os.path.join(dirpath, proto_file)
            command = (
                f'python3 -m grpc_tools.protoc '
                f'--proto_path={dirpath} '  # Use proto_root_dir for --proto_path
                f'--python_out={dirpath} '
                f'--grpc_python_out={dirpath} '
                f'{proto_file_path}'
            )
            subprocess.run(command, shell=True, check=True)
            print("OK!")

        # Create __init__.py in the directory of .proto file
        init_file_path = os.path.join(dirpath, '__init__.py')
        if not os.path.exists(init_file_path):
            with open(init_file_path, 'w') as init_file:
                init_file.write("import os\nimport sys\n\nsys.path.insert(0, os.path.abspath(os.path.dirname(__file__)))\n")
            print(f"Created __init__.py in {dirpath}")

See issue

Collectives™ on Stack Overflow

Python gRPC: Cannot import <generated module> from proto

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related