How to read structured binary data from a file?

Question

The following C++ code writes a header to a file:

#include <iostream>

struct Header
{
    uint16_t name;
    uint8_t type;
    uint8_t padding;
    uint32_t width, height;
    uint32_t depth1, depth2;
    float dMin, dMax;
};

int main()
{

  Header header;
  header.name = *reinterpret_cast<const uint16_t*>("XO");
  header.type = true;
  header.width  = (uint32_t)512;
  header.height = (uint32_t)600;
  header.depth1  = (uint32_t)16;
  header.depth2 = (uint32_t)25;
  header.dMin = 5.0;
  header.dMax = 8.6;

  FILE* f = fopen("header.bin", "wb");
    fwrite(&header, sizeof(Header), 1, f);
}

I am looking to read these header.bin files using Python. In C++ I would be doing something like:

fread(&header, sizeof(Header), 1, f)

But I'm unsure how to read the bytes and convert them into the corresponding fields that the Header struct has in Python?

Python has an aptly named struct module suitable for such tasks. — MisterMiyagi
– MisterMiyagi, Commented Jun 2, 2021 at 17:46

MisterMiyagi · Accepted Answer · 2021-06-03 04:52:25Z

4

Use the struct module to define the binary layout of a C-like struct and de-/serialise it:

import struct

# Format String describing the data layout
layout = "H B x 2L 2L 2f"

# Object representing the layout, including size
header = struct.Struct(layout)

with open("header.bin", "rb") as in_stream:
    print(header.unpack(in_stream.read(header.size))

The layout is a format string describing the fields in-order, e.g. H for uint16_t, B for uint8_t, x for a pad byte, and so on.

edited Jun 3, 2021 at 4:52

answered Jun 2, 2021 at 17:58

MisterMiyagi

53.4k14 gold badges131 silver badges138 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

ti7 Over a year ago

this is probably the cleanest approach

Ben Y Over a year ago

Be sure to precede the layout with the appropriate endianness character if the data can be affected by endianness. (see the documentation for the struct module). Typically < or > for forcing little or big-endianness, respectively

nickponline Over a year ago

Thank you and yes I needed to add the little-endian layout = "<H B x 2L 2L 2f"

Mark Tolonen Over a year ago

FYI ctypes.Structure also supports bit fields, which struct does not, if that is desired.

ti7 · Accepted Answer · 2021-06-04 19:28:34Z

I would do this with Python's ctypes, somewhat so you can share the Header header

Create a class from ctypes.Structure to map the types

import ctypes

class StructHeader(ctypes.Structure):
    _fields_ = [
        ("name", ctypes.c_uint16),
        ("type", ctypes.c_uint8),
        ...
    ]

And create a function which does what you want with a signature like

int header(struct Header &buffer)
{
    // open the file and write to buffer
    // opportunity for other features
}

Then you can compile a shared object to read it which returns that type

gcc -shared -Wl,-soname,your_soname \
    -o library_name file_list library_list

And call out with ctypes.CDLL to read the headers

header = ctypes.CDLL("mylib.so.1").header  # function named header
header.argtypes = [ctypes.POINTER(StructHeader)]
header.restype  = ctypes.c_int

# allocate struct for write
buffer = StructHeader()

# call out to function to write buffer
header(buffer)

Collectives™ on Stack Overflow

How to read structured binary data from a file?

2 Answers 2

4 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related