Write struct with different sizes into binary file and read it

Question

If you worked with postgresql, you may know it stores various stats in internal views. I want to get stats with some interval and store it into a binary file. I see it in the following way:

read stats with PQexec() from libpq.
using info about the number of rows and columns from result, allocate a 3d pointer array.
put result to the 3d array (which is already a part of struct)
write struct with 3d array to binary file
go to step 1 and repeat

Next, it's possible to read this binary file, read struct in to the similar 3d array and then sort values, view and so on.

And everything looks fine, until number of rows is constant, but in postgresql there are views where number of rows are changed permanently. Thus sizes of written structs will be always different and I don't know how to properly read these structs from file (because I need to know the size of struct).

Could you advise me or propose a better way to do it?

It sounds like you just need to write some metadata as a header in your file before the actual data. Then you'll have no problem encoding the element sizes. — John Zwinck
– John Zwinck, Commented Feb 18, 2017 at 5:29
yep, I thought about header and there is a one moment which confuses me -- Header is in the file has fixed size, thus we can put into the file a limited number of stats snapshots. Maybe it's better to user dedicated header for every stats snap. And firstly read row header (where size is located) and then read the date. — lesovsky
– lesovsky, Commented Feb 18, 2017 at 6:18

Caleb · Accepted Answer · 2017-02-18 05:33:09Z

2

Basically you need to adopt a file format. The most basic format for something like this would be to write the size first and then the data. Then the reader would read the size, allocate the memory and then read the data.

answered Feb 18, 2017 at 5:33

Caleb

1,1735 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Jonathan Leffler · Accepted Answer · 2017-02-18 05:34:23Z

0

You're going to need a variant on TLV — Type, Length, Value — encoding. You store (fixed size) information about the variable length data that comes next. When reading, you read the fixed size info and use that to allocate the right space for reading the data that follows.

answered Feb 18, 2017 at 5:34

Jonathan Leffler

759k145 gold badges961 silver badges1.3k bronze badges

Collectives™ on Stack Overflow

Write struct with different sizes into binary file and read it

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related