Why protocol buffer c++ library not reading binary objects properly

Question

I created a binary file using a c++ program using protocol buffers. I had issues reading the binary file in my C# program, so I decided to write a small c++ program to test the reading.

My proto file is as follows

message TradeMessage {
required double timestamp = 1;
required string ric_code = 2;
required double price = 3;
required int64 size = 4;
required int64 AccumulatedVolume = 5;
}

When writing to protocol buffer, I first write the object type, then object length and the object itself.

coded_output->WriteLittleEndian32((int) ObjectType_Trade); 
coded_output->WriteLittleEndian32(trade.ByteSize()); 
trade.SerializeToCodedStream(coded_output);

Now, when I am trying to read the same file in my c++ program i see strange behavior.

My reading code is as follows:

coded_input->ReadLittleEndian32(&objtype);
coded_input->ReadLittleEndian32(&objlen);
tMsg.ParseFromCodedStream(coded_input);
cout << "Expected Size = " << objlen << endl;
cout<<" Trade message received for: "<< tMsg.ric_code() << endl;
cout << "TradeMessage Size = " << tMsg.ByteSize() << endl;

In this case, i get following output

Expected Size = 33
Trade message received for: .CSAP0104
TradeMessage Size = 42

When I write to file, I write trade.ByteSize() as 33 bytes, but when I read the same object, the object ByteSize() is 42 bytes, which affects the rest of the data. I am not sure what is wrong in this. Please advice.

Regards, Alok

just to double check.. I compared the protocol buffer generated files in my reader and writer projects. The generated files are identical. So I guess, the file coding is different for some reason. I do not understand why it is different. — Alok
– Alok, Commented Dec 9, 2011 at 6:01
Any chance you can share your test values, just so I can check which is "more right" ? — Marc Gravell
– Marc Gravell, Commented Dec 9, 2011 at 6:57
also - minor point; if you wrote ([objtype] << 3) | 2 (as varint), then the byte-size (as varint), then the raw data, it would actually be a valid protobuf sequence itself, which can be useful - just sayin' — Marc Gravell
– Marc Gravell, Commented Dec 9, 2011 at 7:01
@MarcGravell can I mail you the binary data file? I am afraid there is no other way i can send you the file from here. I will make the changes regarding objtype (and some other fields) as you suggested. — Alok
– Alok, Commented Dec 9, 2011 at 7:26

Marc Gravell · Accepted Answer · 2011-12-09 08:14:07Z

1

This is guesswork, based on the above: when you use ParseFromCodedStream, you aren't actually limiting that to the objlen that you previously found; thus, if the stream contains any more data than this (i.e. that isn't the end of the file), the engine will try to keep reading to the EOF. You must cap the length to your expectation. I am not a C++ expert, so I can't offer direct guidance, but if this was C# (using protobuf-net):

objType = ProtoReader.DirectReadLittleEndianInt32(file);
len = ProtoReader.DirectReadLittleEndianInt32(file);

// assume GetObjectType returns typeof(TradeMessage) for our objType
Type type = GetObjectType(objType);
msg = RuntimeTypeModel.Default.Deserialize(file, null, type, len, null);

answered Dec 9, 2011 at 8:14

Marc Gravell

1.1m273 gold badges2.6k silver badges3k bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Alok Over a year ago

apologies for late reply. I was travelling. SO i used your tip and put the specified the lengths while reading. Later I found this in the documentation itself. The idea is to push the length to read in the stream. c++ code is ok. But I still have the same issue while reading from my C# code :( using above code to read the objects, it still stops at the same location with following error .. ProtoBuf.ProtoException: Incorrect number of bytes consumed at ProtoBuf.Meta.TypeModel.Deserialize(Stream source, Object value, Type type, Int32 length, SerializationContext context)

Alok Over a year ago

for c# version I was using following code TradeMessage tMsg = Serializer.DeserializeWithLengthPrefix<TradeMessage>(stream, PrefixStyle.Fixed32); This should have worked originally but it wasn't working. After this i wrote a test c++ program to check if generated file was ok. C3 version is still isn't working.

Alok Over a year ago

so looks like, even if I specify the length as the argument in function call msg = RuntimeTypeModel.Default.Deserialize(file, null, type, len, null); internally, protobuf-net is not able to read length number of bytes. What could be the reason for this? c++ library is able to read the objects properly. Thanks.

Alok · Accepted Answer · 2011-12-16 05:28:57Z

1

So apparently, i was doing a very silly mistake while creating the binary files. I did not open the file in binary mode when i wrote protobuf data to it causing it to add weird ascii characters in the middle. This caused an issue while reading the data using protobuf-net library. The issue is resolved here. Shouldn't have taken so long to resolve this.

answered Dec 16, 2011 at 5:28

Alok

3,3403 gold badges32 silver badges47 bronze badges

Collectives™ on Stack Overflow

Why protocol buffer c++ library not reading binary objects properly

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related