I need to save a large 3D array of integers into a file, and load it again in C++. It is 256*256*256 = 16777216 integers.
What is the best way to save this and load it again? I am mostly interested in a quick load time.
-
You should be able to load from the file using multiple threads (as any number of readers are fine). Storing them in the same array has to be done carefully (so as to not interfere with each other). I would try and save the data in such a way as to make multiple threads feasible (i.e. a line break format).twain249– twain2492012-04-11 20:41:34 +00:00Commented Apr 11, 2012 at 20:41
-
1@twain249 I'd say the bottleneck is the disk, adding to seek times by moving it back and forth within the file won't make it better by much... Just sayin...oh whatever– oh whatever2012-04-11 20:44:09 +00:00Commented Apr 11, 2012 at 20:44
-
Maybe see question How to compress a buffer with zlibOpen AI - Opting Out– Open AI - Opting Out2012-04-11 20:54:19 +00:00Commented Apr 11, 2012 at 20:54
Add a comment
|
2 Answers
If the array is allocated in contiguous memory (i.e.: you don't allocate each dimension separately) - you can just dump the whole memory block to file. It takes as much as it takes, but that would be the least overhead (i.e.: call binary write on the whole chunk of data).
If you're saving on one system and loading on another, you might have issues with data representation, in this case you'd probably want to serialize the array and save each value in a controlled matter.
1 Comment
hansdam
Thanks! I'll try to implement that. Got any web references that could help me?
You may be interested in Boost.Serialization, particularly if you (1) want the ability to easily store such data on disk, (2) want a coherent way to save more complex objects, and (3) want a solution that's portable.
