How to read a binary file with a 32-bit float, a 64-bit float and another-32 bit float in Python

Question

This is a binary file with a very simple structure for learning purposes. Each register has 3 numbers: a 32-bit float, a 64-bit float and another 32-bit float. If I dump it on the screen in hexadecimal, it looks like this:

0000000: 0800 0000 0000 0000 0000 0000 0800 0000  ................
0000010: 0800 0000 0000 0000 0000 f03f 0800 0000  ...........?....
0000020: 0800 0000 182d 4454 fb21 0940 0800 0000  .....-DT.!.@....

(...)

If I manually copy the third line in binary format, I can read it into three variables:

import struct
data = b'\x08\x00\x00\x00\x18-DT\xfb!\t@\x08\x00\x00\x00'
l1, value, l2 = struct.unpack("<idi", data)
# (8, 3.141592653589793, 8)

That works, but I need to read the file from disk, not only manually copying each register in binary, because I need to do this with millions data. I need something equivalent to the following command used in ascii files:

l1, value, l2 = pylab.loadtxt('./test_file.binary',unpack=True)

Which doesn't work here.

is this the entire file or is there more data in the file? If there is more then what do you expect to get back into your three variables? — Tadhg McDonald-Jensen
– Tadhg McDonald-Jensen, Commented Apr 26, 2017 at 19:28
There are several million registers (or lines or however you name it) per file. — Mephisto
– Mephisto, Commented Apr 26, 2017 at 19:28

wim · Accepted Answer · 2017-04-26 23:24:57Z

3

Read the file in binary mode:

def read_stuff(fname='test_file.binary'):
    with open(fname, mode='rb') as f:
        while True:
            data = f.read(16)
            if len(data) < 16:
                # end of file
                return
            yield struct.unpack("<idi", data)

This is a generator. To consume it:

for l1, value, l2 in read_stuff():
    ...

edited Apr 26, 2017 at 23:24

answered Apr 26, 2017 at 19:31

wim

368k114 gold badges682 silver badges818 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Bill Bell Over a year ago

Missing if in that one statement?

wim Over a year ago

@BillBell fixed, thanks. Feel free to edit directly next time.

Mephisto Over a year ago

def read_stuff(fname='test_file.binary') SyntaxError: invalid syntax and it points at the closing parenthesis. Cryptic python error messages. Shouting to the screen after two hours after this. It should be simple.

Denis · Accepted Answer · 2019-04-26 02:58:51Z

You may try this method

# includes core parts of numpy, matplotlib
import matplotlib.pyplot as plt
import numpy as np
# include scipy's signal processing functions
import scipy.signal as signal


# practice reading in complex values stored in a file
# Read in data that has been stored as raw I/Q interleaved 32-bit float samples
dat = np.fromfile("iqsamples.float32", dtype="float32")
# Look at the data. Is it complex?


dat = dat[0::2] + 1j*dat[1::2]
print(type(dat))
print(dat)

# # Plot the spectogram of this data
plt.specgram(dat, NFFT=1024, Fs=1000000)
plt.title("PSD of 'signal' loaded from file")
plt.xlabel("Time")
plt.ylabel("Frequency")
plt.show()  # if you've done this right, you should see a fun surprise here!

Collectives™ on Stack Overflow

How to read a binary file with a 32-bit float, a 64-bit float and another-32 bit float in Python

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related