Strange issue serializing array of DateTime and back

Question

I am experiencing a strange issue when serializing a large array of DateTime (each timestamp is of Kind UTC). I proceeded with the following steps:

convert each DateTime to type long via DateTime.ToBinary()
convert long[] to byte[] via Buffer.BlockCopy
write the byte array to a file stream
read in the byte array via file stream
convert byte[] to long[] via Buffer.BlockCopy
convert each long back to DateTime via DateTime.FromBinary(long)

The issue is that the DateTimes are not matching between the original array and the final array. In fact some of the time stamps show as year 2059 or so when the original array strictly contained time stamps of the past.

I run the entire procedure on my local machine in Windows 10 hence there should be no time zone issues nor issues of endianess. Can someone help?

This is how I convert time stamps of type DateTime to long[]:

var dataCollection = new DataCollection(header.DataProviderId, DateTimeKind.Utc, header.Symbol, header.QuoteType, header.Compression)
        {
            TimeStamps = quotes.Select(x => x.TimeStamp.ToBinary()).ToArray(),
            Bid = quotes.Select(x => x.Bid).ToArray(),
            Ask = quotes.Select(x => x.Ask).ToArray()
        };

Here are the conversions between long[] -> byte[] and back:

public static byte[] SerializeBlockCopy<T>(Array sourceArray, long sourceStartIndex, long numberItemsToSerialize)
    {
        var targetArraySize = numberItemsToSerialize * Marshal.SizeOf(typeof(T));
        var targetArray = new byte[targetArraySize];
        Buffer.BlockCopy(sourceArray, (int)sourceStartIndex, targetArray, 0, (int)targetArraySize);
        return targetArray;
    }

    public static T[] DeserializeBlockCopy<T>(byte[] sourceArray)
    {
        var targetArraySize = sourceArray.Length / Marshal.SizeOf(typeof(T));
        var targetArray = new T[targetArraySize];
        Buffer.BlockCopy(sourceArray, 0, targetArray,0, sourceArray.Length);
        return targetArray;
    }

@Oliver, let me add some code snippets but I can't post the entire library. — Matt
– Matt, Commented Jan 22, 2020 at 6:23

Oliver · Accepted Answer · 2020-01-22 06:53:50Z

1

I can't spot any errors. Here is an example program I tested with your methods and they worked as expected:

public static void Main()
{
    var random = new Random();
    var sourceDates = Enumerable.Range(1, 100)
        .Select(i => DateTime.UtcNow.Add(TimeSpan.FromDays(random.Next(-1000, 1000))))
        .ToList();

    var values = sourceDates.Select(date => date.ToBinary()).ToArray();
    var asBytes = SerializeBlockCopy(values, 0, values.Length);
    var filename = Path.GetTempFileName();
    WriteToFile(asBytes, filename);

    var bytesFromFile = ReadFromFile(filename);
    var back = DeserializeBlockCopy<long>(bytesFromFile);
    File.Delete(filename);
    var destinationValues = back.Select(value => DateTime.FromBinary(value)).ToList();
    var pairs = sourceDates.Zip(destinationValues, (s, d) => (s, d));

    foreach (var pair in pairs)
    {
        Console.WriteLine($"{pair.s} -> {pair.d}");
    }

    Console.WriteLine($"Both are equal: {sourceDates.SequenceEqual(destinationValues)}");
}

public static void WriteToFile(byte[] source, string filename)
{
    using (var writer = new FileStream(filename, FileMode.Truncate, FileAccess.Write))
    {
        writer.Write(source, 0, source.Length);
    }
}

public static byte[] ReadFromFile(string filename)
{
    return File.ReadAllBytes(filename);
}

For a slight improvement you could change the signature of your serialize method to:

public static byte[] SerializeBlockCopy<T>(T[] sourceArray, long sourceStartIndex, long numberItemsToSerialize)

Then you don't need to provide the generic argument, cause type inference jumps in, but that wouldn't explain your error.

edited Jan 22, 2020 at 6:53

answered Jan 22, 2020 at 6:33

Oliver

45.4k8 gold badges101 silver badges165 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Matt Over a year ago

I tried that already and just serializing and deserializing on the fly works fine. But somehow when it goes to disk and back something strange is going on.

Oliver Over a year ago

Maybe you should check that code and make a byte comparison of the written and read bytes like in the example above. If you add theses methods to the above example you could quite easily also check the input and output of those methods too.

Oliver Over a year ago

Just added some code to write to and read from a file on disk and no error occurs.

Matt Over a year ago

I noticed that there are no errors on my side either when reading the entire file. So, I guess the error must be residing in my routine to find a start and end time stamp on file and only reading that chunk. Thank you for your help, I could reduce the potential error to the finding of a time stamp directly in the random access binary file.

Matt Over a year ago

Found a nasty bug in the Buffer.BlockCopy method. Even the offsets have to be expressed in byte-length not element length when dealing with other primitives in the source array.

Collectives™ on Stack Overflow

Strange issue serializing array of DateTime and back

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related