Protobuf 3.19.1 and Python 3.10 issue

Question

This is all about and issue when using the latest Python Protobuf (3.19.1) and Python 3.10, in Linux (tested in Fedora 35 and Ubuntu 20.04.

It broke our library but it can easily tested using the addressbook.proto from the Python Protobuf tutorial and tried to get the proto2 message class as follows:

import addressbook_pb2

from google.protobuf import (
    descriptor_database,
    descriptor_pb2,
    descriptor_pool,
    message_factory,
)

_DESCRIPTOR_DB = descriptor_database.DescriptorDatabase()
_DESCRIPTOR_POOL = descriptor_pool.DescriptorPool(_DESCRIPTOR_DB)

_DESCRIPTOR_DB.Add(
    descriptor_pb2.FileDescriptorProto.FromString(
        addressbook_pb2.DESCRIPTOR.serialized_pb
    )
)
factory = message_factory.MessageFactory()
cls = factory.GetPrototype(_DESCRIPTOR_POOL.FindMessageTypeByName("tutorial.Person"))

It raises the following error:

[libprotobuf ERROR google/protobuf/pyext/descriptor_database.cc:64] DescriptorDatabase method raised an error
SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats
Traceback (most recent call last):
  File "/dev/protobuf/test/test.py", line 21, in <module>
    ls = factory.GetPrototype(_DESCRIPTOR_POOL.FindMessageTypeByName("tutorial.Person"))
`KeyError: "Couldn't find message tutorial.Person"

Now, it works as expected if I use an older Python Protobuf version, such as 3.18.1.

I've opened a bug https://github.com/protocolbuffers/protobuf/issues/9245, but apparently, it was not considered a bug.

Python Protobuf introduced the PY_SSIZE_T_CLEAN macro in 3.19.1 and broke something, probably by using int instead of Py_ssize_t when using # formats.

Have anyone have this issue or can confirm it?

jpa · Accepted Answer · 2022-06-03 07:08:05Z

2

There are actually two errors here:

SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats

This error is caused by Python 3.10 dropping support for old default conversions when passing data from C to Python side. In this case in the protobuf library, the error only occurs when passing an exception from C code to Python.

The python-protobuf library was fixed to work with Python 3.10 back in October 2021, and the fix should be included in python-protobuf 3.20.0 and later.

Try adding this to your script to check the version:

import google.protobuf
print(google.protobuf.__version__)

For me the error does not occur with the latest versions 3.19.4, 3.20.1 or 4.21.1, but does occur with 3.19.2 and older.

answered Jun 3, 2022 at 7:08

jpa

12.5k1 gold badge40 silver badges57 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Divya Nambiar · Accepted Answer · 2022-06-03 06:34:28Z

1

Yes I am also getting same issue.

We changes the version as below:

*protobuf >= 3.19* # Not working

*protobuf >= 3.15.6, <= 3.20.1* # Working

edited Jun 3, 2022 at 6:34

answered Jun 3, 2022 at 6:29

Divya Nambiar

1791 silver badge3 bronze badges

Collectives™ on Stack Overflow

Protobuf 3.19.1 and Python 3.10 issue

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related