How to generate python class files from protobuf

Question

I am trying to transfer large amounts of structured data from Java to Python. That includes many objects that are related to each other in some form or another. When I receive them in my Python code, it's quiet ugly to work with the types that are provided by protobuf. My VIM IDE crashed when trying to use autocomplete on the types, PyCharm doesn't complete anything and generally it just seems absurd that they don't provide some clean class definition for the different types.

Is there a way to get IDE support while working with protobuf messages in python? I'm looking at 20+ methods handling complex messages and without IDE support I might as well code with notepad.

I understand that protobuf is using metaclasses (although I don't know why they do that). Maybe there is a way to generate python class files from that data or maybe there is something similar to typescript typing files.

Did I maybe misuse protobuf? I believed I would describe my domain model in a way that may be used across languages. In Java I am happy with the generated classes and I can use them easily. Should I maybe have used something like swagger.io instead?

Daniel · Accepted Answer · 2019-10-31 04:38:22Z

22

If you are using a recent Python (3.7+) then https://github.com/danielgtaylor/python-betterproto (disclaimer: I'm the author) will generate very clean Python dataclasses as output which will give you proper typing and IDE completion support.

For example, this input:

syntax = "proto3";

package hello;

// Greeting represents a message you can tell a user.
message Greeting {
  string message = 1;
}

Would generate the following output:

# Generated by the protocol buffer compiler.  DO NOT EDIT!
# sources: hello.proto
# plugin: python-betterproto
from dataclasses import dataclass

import betterproto


@dataclass
class Hello(betterproto.Message):
    """Greeting represents a message you can tell a user."""

    message: str = betterproto.string_field(1)

In general the output of this plugin mimics the *.proto input and is very easy to read if you happen to jump to definition on a message or field. It's been a huge improvement for me personally over the official Google compiler plugin, and supports async gRPC out of the box as well.

answered Oct 31, 2019 at 4:38

Daniel

8,7902 gold badges47 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

moshevi Over a year ago

Why does betterproto uses custom fields for data classes instead of python's built-in field ?

Daniel Over a year ago

@moshevi internally it is actually using dataclasses.field. The field functions are just shortcuts. See github.com/danielgtaylor/python-betterproto/blob/…

moshevi Over a year ago

but this way you are trying my code to protobuff. It would make sense to have a data access layer Wich outs technology agnostic python types and dataclasses

Roshin Raphel · Accepted Answer · 2023-03-19 09:51:26Z

9

What "-Zaffy" said worked for me. (--pyi_out)

EXAMPLE

Note : Protobuf filename example = addressbook.proto

CLI Cmd to create intellsense info (compiles in this same dir)

protoc -I . --pyi_out=. addressbook.proto

Then create python code. (compiles in this same dir)

protoc -I . --python_out=. addressbook.proto

This will create a addressbook_pb2.py and addressbook_pb2.pyi

then VS code away!

edited Mar 19, 2023 at 9:51

Roshin Raphel

2,7195 gold badges28 silver badges43 bronze badges

answered Mar 14, 2023 at 14:15

UCF10

911 silver badge2 bronze badges

1 Comment

Bhakta Raghavan Over a year ago

This is the best answer! I am surprised it has no upticks!

tuk · Accepted Answer · 2018-09-23 07:22:56Z

3

mypy-protobuf generates the type hint files. But as discussed here this works only from protobuf 3.0 and python 2.7 onwards.

answered Sep 23, 2018 at 7:22

tuk

6,95422 gold badges101 silver badges210 bronze badges

Comments

Michał · Accepted Answer · 2018-04-10 17:54:46Z

3

As of now, nothing like that is available. You might want to follow this issue: https://github.com/google/protobuf/issues/2638 to be up to date.

answered Apr 10, 2018 at 17:54

Michał

2,2923 gold badges22 silver badges33 bronze badges

4 Comments

pascalwhoop Over a year ago

While I didn’t find a solution, I did find this which generates type hint files github.com/dropbox/mypy-protobuf

tuk Over a year ago

@pascalwhoop - Does mypy-protobuf works with protoc 2.5.0 and python 2.7.6 ?

pascalwhoop Over a year ago

@tuk not sure about the exact versions but it did work for my project.

user823738 Over a year ago

The issue is already resolved and the new options is --pyi_out=. Perhaps you could update your answer. I find it far more better than the others.

Collectives™ on Stack Overflow

How to generate python class files from protobuf

4 Answers 4

3 Comments

What "-Zaffy" said worked for me. (--pyi_out)

EXAMPLE

1 Comment

Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

3 Comments

What "-Zaffy" said worked for me. (--pyi_out)

EXAMPLE

1 Comment

Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related