Python mangles struct.pack strings written to disk

Question

I'm working on an application writing binary data (ints, doubles, raw bytes) to a file.

Problem is, that the data is not actually written to the file the way I expect it to be:

>>> import struct
>>> import io
>>> out = io.open("123.bin", "wb+")
>>> format = "!i"
>>> data = struct.pack(format, 1)
>>> out.write(data)
4L
>>> data
'\x00\x00\x00\x01'
>>> out.close()
>>> infile = io.open("123.bin", "rb")
>>> instr = infile.read()
>>> instr
'\x00\x00\x00\x01'
>>> struct.unpack("!I", instr)
(1,)

So everything looks like it's working just fine. But upon closer examination, the 123.bin file has following contents:

$ hexdump 123.bin 
0000000 0000 0100                              
0000004

So it looks like the bytes were swapped by io.write()!

The python documentation says, that io.write() accepts "given bytes or bytearray object", problem is, that struct.pack does return an str:

>>> type(struct.pack(format, 1))
<type 'str'>

So, what am I doing wrong? How do I convert str to bytes without any charset translation?

Aya · Accepted Answer · 2013-06-27 17:27:15Z

4

Looks like this is an oddity of hexdump(1). Using xxd(1), I get...

$ xxd 123.bin
0000000: 0000 0001                                ....

...which looks correct.

Looks like you have to use the -C option to get hexdump(1) to output in a sane format...

$ hexdump -C 123.bin
00000000  00 00 00 01                                       |....|
00000004

...or call it as hd instead.

answered Jun 27, 2013 at 17:27

Aya

42.5k6 gold badges60 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Alicja Kario Over a year ago

hexdump -C returns 00000000 00 00 00 01 |....|, so yeah, it just looks like the default mode of hexdump is "broken".

Simon Callan · Accepted Answer · 2013-06-27 17:32:30Z

1

The problem here isn't with python, but with hexdump. It's treating the data in the file as 16 bit little endian values. What you need to do is tell hexdump to treat the data as 8 bit values. Without looking it up, I think it's the '-c' option.

answered Jun 27, 2013 at 17:32

Simon Callan

3,1401 gold badge24 silver badges35 bronze badges

Comments

Sylvain Defresne · Accepted Answer · 2013-06-27 17:34:22Z

The default output format of hexdump is the same as using the -x option, that is, according to the man page:

 -x      Two-byte hexadecimal display.  Display the input offset in hexadecimal,
         followed by eight, space separated, four column, zero-filled, two-byte
         quantities of input data, in hexadecimal, per line.

And the endianness used by hexdump is the architecture endianness (here likely little-endian), while you asked python to store the value in network order (big-endian).

So, the value is correctly stored, but incorrectly interpreted by hexdump. You can either use the -C option or xxd instead of hexdump.

$ hexdump 123.bin
0000000 0000 0100                              
0000004
$ hexdump -C 123.bin
00000000  00 00 00 01                                       |....|
00000004

Collectives™ on Stack Overflow

Python mangles struct.pack strings written to disk

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related