ProtoBuf-Net error message " Invalid field in source data: 0"

Question

I succeed in serializing instances of the following class but when I try to deserialize right after I get the following error message: " Invalid field in source data: 0".

I have no clue what it refers to because I find below class straight forward. I just updated protobuf-net version to 2.00.614 (runtime version: 2.0.50727).

Any idea whether I am possibly overlooking something trivial?

[ProtoContract]
public class TimeSeriesProperties 
{
    [ProtoMember(1)]
    public string TimeSeriesName { get; private set; }
    [ProtoMember(2)]
    public string FileName { get; private set; }
    [ProtoMember(3)]
    public string TemplateName { get; private set; }
    [ProtoMember(4)]
    public int PacketLength { get; private set; }
    [ProtoMember(5)]
    public long FileSizeBytes { get; set; }
    [ProtoMember(6)]
    public long NumberRecords { get; set; }
    [ProtoMember(7)]
    public DateTime DateTimeStart { get; set; }
    [ProtoMember(8)]
    public DateTime DateTimeEnd { get; set; }

    public TimeSeriesProperties()
    {

    }

    public TimeSeriesProperties(string timeSeriesName, string fileName, string templateName, int PacketLength)
    {
        this.TimeSeriesName = timeSeriesName;
        this.FileName = fileName;
        this.TemplateName = templateName;
        this.PacketLength = PacketLength;
    }

}

public static byte[] Serialize_ProtoBuf<T>(T serializeThis)
    {
        using (var stream = new MemoryStream())
        {
            ProtoBuf.Serializer.Serialize<T>(stream, serializeThis);
            return stream.GetBuffer();
        }
    }

    public static T Deserialize_ProtoBuf<T>(byte[] byteArray)
    {
        using (var stream = new MemoryStream(byteArray))
        {
            return ProtoBuf.Serializer.Deserialize<T>(stream);
        }
    }

That almost always means you are over-reading the written data, typically by incorrect handling of MemoryStream - nothing to do with protobuf-net; please can you show the code that does the serialize/deserialize test? (and ideally, add a @marc comment, so I know to come back and look at it) — Marc Gravell
– Marc Gravell, Commented Jan 7, 2013 at 13:29
that approach would never have worked with any protobuf implementation; it is a feature of the specification that the outermost message does not know its own length - this is so that fragments are mergeable via concatenation. Thus, by default it reads to the end of the stream (although most implementations include a mechanism for reading n bytes, for some n). At the end of each field, it expects either another field-header, or an EOF. 0 is never a valid field-header, so trailing zeros will break the deserializer. Every time. — Marc Gravell
– Marc Gravell, Commented Jan 7, 2013 at 13:44
incidentally, the amount of data written is simply: stream.Length — Marc Gravell
– Marc Gravell, Commented Jan 7, 2013 at 13:48
that depends entirely on the context; in the scenario you show (with most Streams etc), the length is trivially available, and the entire thing is fixed just via ToArray(), or a constrained read. With some implementations (such as a NetworkStream sending multiple messages), [Serialize|Deserialize]WithLengthPrefix is your friend. I'm not sure why you don't know the length after serializing. Can you expand on that? If this is for a header at the start of a file, the *WithLengthPrefix should work fine. — Marc Gravell
– Marc Gravell, Commented Jan 7, 2013 at 13:55

Marc Gravell · Accepted Answer · 2013-01-07 13:40:46Z

11

The most common cause I've seen of this is simply incorrect use of GetBuffer() on a MemoryStream. That was already my hunch when I added my comment, but you've confirmed it:

using (var stream = new MemoryStream())
{
    ProtoBuf.Serializer.Serialize<T>(stream, serializeThis);
    return stream.GetBuffer();
}

GetBuffer returns the oversized backing-buffer. It has garbage at the end of it. It is perfectly fine to use GetBuffer, as long as you also record the .Length, which is the amount of valid data in there. This can avoid an extra allocation of a potentially large array. But in your case, a simpler approach is probably to use ToArray() to get a right-sized buffer:

using (var stream = new MemoryStream())
{
    ProtoBuf.Serializer.Serialize<T>(stream, serializeThis);
    return stream.ToArray();
}

answered Jan 7, 2013 at 13:40

Marc Gravell

1.1m273 gold badges2.6k silver badges3k bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Matt Over a year ago

Marked as answered. The key for me was the "WithLengthPrefix" issue. I could have sworn this worked without before, my code broke after I updated to the current ProtoBuf-net version. I can assure you that I serialized to a byte array of size 256 even though the actual serialized object was only of length 146 and I deserialized by passing in a 10,000 byte array where only the first 146 bytes were taken up by the serialized object. It worked, I am absolutely sure of that. I have no clue what changed but I know my code was not safe because the serialized object was not prefixed with size.

Etienne Over a year ago

This solution works if you are working with a fresh archive. Is there a way to fix this issue if we already have archives were written out using GetBuffer() and we no longer know the size of the archive?

Marc Gravell Over a year ago

@Etienne You could potentially use the reader API to check for when the next field-header is zero, which is never legal - only works if the MemoryStream was not previously used for something else random (i.e. is the unused space all zeros?). Heck, you could probably just remove all the trailing zeros - there's an edge case that there were legitimate trailing zeros, but it is easier than hooking the reader API. What sort of quantity are you talking about here? 10? 1000? 1000000?

Etienne Over a year ago

@Marc, I ended up removing the trailing zeros and it worked for me. It's probably not the most ideal solution but in my case this will be executes so infrequently that it doesn't really matter. I was hoping that the library had some cool way of dealing with it but I was wrong.

Collectives™ on Stack Overflow

ProtoBuf-Net error message " Invalid field in source data: 0"

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related