Deserialisation to Protobuf object in Python

Question

I have passed a Protobuf object in a Kafka producer and am receiving a byte array on the consumer side. Now I want to deserialize that response again back to a Protobuf object, but I am unable to do that. How can I do that?

Here is my consumer:

from email import message
from kafka import KafkaConsumer
import json
from simple import simple_message
import check_pb2

consumer = KafkaConsumer('latest', api_version=(0, 10, 1),
                         group_id='my-group', enable_auto_commit=False,
                         bootstrap_servers=['localhost:9092'])#,
           #value_deserializer=lambda m: json.loads(m.decode('ascii')))
for message in consumer:
    print(message)

I tried to use Confluent but getting

Kafka Exception : {} KafkaError{code=_VALUE_DESERIALIZATION,val=-159,str="Unknown magic byte. This message was not produced with a Confluent Schema Registry serializer"

OneCricketeer · Accepted Answer · 2022-10-31 21:50:04Z

1

If you're not using Confluent Schema Registry, use ParseFromString.

for message in consumer:
    value = ParseFromString(message.value)  # example 
    print(value)

If (and only if) you are using the Schema Registry, then Confluent has a Protobuf deserializer class. See the example, which happens to also use ParseFromString internally.

edited Oct 31, 2022 at 21:50

answered Jul 11, 2022 at 11:55

OneCricketeer

193k20 gold badges146 silver badges276 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

Ashu mishra Over a year ago

tried to use confulent but getting Kafka Exception : {} KafkaError{code=_VALUE_DESERIALIZATION,val=-159,str="Unknown magic byte. This message was not produced with a Confluent Schema Registry serializer"

OneCricketeer Over a year ago

Your producer would need to use the registry as well.

Ashu mishra Over a year ago

my producer is a java service and i do not have control over it

Ashu mishra Over a year ago

can you tell me some way to do that

Ashu mishra Over a year ago

my java part has no schema regitry, can you tell me how can i do that without confluent deserializer ,/

|

Collectives™ on Stack Overflow

Deserialisation to Protobuf object in Python

1 Answer 1

8 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Related