Save byte-arrays to disk to reduce memory consumption and increase speed?

Question

I have a few thousand of very big radio-telemetry array fields of the same area in a database. The georeference of the pixels is the same for all of the array fields. An array can be loaded into memory in an all or nothing approach.

I want to extract the pixel for a specific geo-coordinate from all the array fields. Currently I query for the index of the specific pixel for a specific geocoordinate and then load all array fields from the database into memory. However that is very IO intensive and overloads our systems.

I'd imagine the following: I save the arrays to disk and then sequentially open them and seek to the byte-position corresponding to the pixel. I imagine that this is far less wasteful and much speedier than loading them all to memory.

Is seeking to a position considered a fast operation or would one not do such a thing?

Seeks are generally fast so you should have no trouble. Do you hit the swap when you do it in memory? — dmg
– dmg, Commented Feb 18, 2013 at 10:26
IF possible (so long as your memory pattern is the same as your disk image) It may be simpler and better performance to memory map the file — camelccc
– camelccc, Commented Apr 10, 2013 at 20:15

scanny · Accepted Answer · 2013-02-18 10:59:42Z

2

The time it takes for a seek operation would be measured in low milliseconds, probably less than 10 in most cases. So that wouldn't be a bottleneck.

However, if you have to retrieve and save all the records from the database either way, you may end up with roughly the same IO load and perhaps greater. The IO time for writing a file is certainly greater than reading into memory.

Time for a small-ish experiment :) Try it with a few arrays and time the performance, then you can do the math to see how it would scale.

answered Feb 18, 2013 at 10:59

scanny

29.5k6 gold badges64 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

AME Over a year ago

I would write once for all pixels and then read many many times individual pixel-coordinates.

Collectives™ on Stack Overflow

Save byte-arrays to disk to reduce memory consumption and increase speed?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related