Creating ProtoBuf object schema for Kafka usage

Question

I am using ProtoBuf over Kafka. We are creating a lot of ProtoBuf messages, but we are stuck on these two options. Wondering what is the best way to do this? Hopefully question won't be downgraded as it is somehow opinion based!

Say, I want to send Employee message on Kafka. I want to use "id" as the kafka key and value as the "Employee" message as below.

Option - 1 includes "id". In this option, your entire object is in one place as a domain object.

message Employee {
 string id;
 string name;
 string title;
}

Option - 2 doesn't include id - as it tries to use Kafka key (which we are populating with id) and let the consumer piece them together when they take a message out of Kafka?

message Employee {
 string name;
 string title;
}

Any suggestions on the pros and cons of these two and any known best practice? Thanks.

Natalia · Accepted Answer · 2020-08-12 19:53:54Z

1

From my experience:

Message should contain all required information.
Key is auxiliary information for the messaging system to know how to work with the message.

So, between two options, the first one is preferable.

answered Aug 12, 2020 at 19:53

Natalia

4,5322 gold badges27 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Creating ProtoBuf object schema for Kafka usage

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related