Different ways to convert a byte into integer in python

Question

from binascii import unhexlify
import time
import struct

var = 'FF'
bytes = unhexlify(var)

start_time = time.time()
for i in range(10000):
  temp = struct.unpack('B', bytes)[0]
print("- %s milli sec -" % ((time.time() - start_time)*1000))


start_time = time.time()
for i in range(10000):
  temp = int.from_bytes(bytes, byteorder='big')
print("- %s milli sec -" % ((time.time() - start_time)*1000))


start_time = time.time()
for i in range(10000):
  temp = bytes[0]
print("- %s milli sec -" % ((time.time() - start_time)*1000))

output

- 5.327939987182617 milli sec -
- 12.086629867553711 milli sec -
- 1.882314682006836 milli sec -

Obviously, the 3rd one is a lot faster than others.

Is there any technical reasoning for this. Also, can someone tell me the pros and cons of these approaches? If there is any other better way to achieve this, please explain it as an answer.

program available here: https://repl.it/repls/AcceptableCompatiblePrograms

Why woudn't the last one be fastest? There's no attribute lookup or call overhead, it's just indexing the array. — jonrsharpe
– jonrsharpe, Commented May 5, 2018 at 20:26
You really shouldn't be using time.time to benchmark things; that's what timeit is for. — abarnert
– abarnert, Commented May 5, 2018 at 20:27
You also shouldn't be doing this if you're trying to convert anything larger than FF to integer. For example, how would you convert 'CAFEBABE'? You should instead use int(var, 16) if you're dealing with hex numbers. — cs95
– cs95, Commented May 5, 2018 at 20:29
Anyway, notice that the unhexlify part takes more time than accessing the bytes—e.g., on my laptop, it's 209ns to unhexlify a single byte, and then only 16ns to access the first element of it. So you may be focusing on the wrong thing in the first place. (Also, is a 16ns operation something you need to worry about optimizing here? If you're doing it billions of times, you may want to step back and find a higher-level way of doing whatever you're actually doing.) — abarnert
– abarnert, Commented May 5, 2018 at 20:30
By comparison, int(var, 16) takes 233ns, which is only 8ns more than unhexlify(var)[0]. Even if that 4% performance cost in the single-digit-nanos range is consistent, is that worth the added brittleness? — abarnert
– abarnert, Commented May 5, 2018 at 20:33

abarnert · Accepted Answer · 2018-05-05 21:31:59Z

1

The first problem is that you're not benchmarking properly (you should be using timeit, because it takes care of all the things you didn't think of), and, even more importantly, you're benchmarking the wrong thing.

Using %timeit in IPython on my laptop, here are the times for the three parts of your process:

b = unhexlify(var): 209ns
b[0]: 16.0ns
temp = b0: 50.2ns

So, you're focusing on the fastest part of the process.

Meanwhile, once you've decided to use unhexlify to get a bytes object, of course the fastest way to get the first byte out of that is b[0]. How could anything possibly be any faster than that?

But if you take a step back:

int(var, 16): 233ns

This is nearly as fast as unhexlify(var)[0]—within 4%, and a difference in the single-digit nanos. Which may not even be consistently repeatable across systems, Python versions, or input values. But, even if it were, it's hard to imagine an application where this 8ns makes a difference where you couldn't get a much bigger speedup by stepping back and doing something at a higher level. Sure, it's not impossible this could come up, by immediately jumping to how to micro-optimize this operation is almost always going to be a mistake.

Even more importantly, unhexlify(var)[0] only works for single-byte values. Try it with, say, FF22 and you're going to get 255 instead of 65314. The other options—including int—will give you the right answer.

And of course using struct and int.from_bytes give you more flexibility—you can read bytes in either endianness, or specify the exact size you expect (or read exactly that many bytes out of an input buffer without consuming the whole buffer).

So, the right answer is to use the one that does what you actually want in the most obvious way. Something that's faster but wrong is not helpful, and even something that's faster but not obviously right often isn't helpful.

And this means that if what you're actually trying to do is (contrary to what you implied in your question) iterate or index a bytes as integers from 0 to 255, the right thing to do is for by in b: or b[0]. Not because it's fastest (although it is), but because it directly does exactly what you want to do—it's the One Obvious Way To Do It.

edited May 5, 2018 at 21:31

answered May 5, 2018 at 20:42

abarnert

368k54 gold badges626 silver badges691 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

lch Over a year ago

I am reading a byte stream. So int(var, 16) is not useful in my case. because var should be hex string

abarnert Over a year ago

@LokeshCherukuri int(var, 16) works on a hex string. Have you tried it?

lch Over a year ago

yes, it works on hex string. but i am dealing with bytes

abarnert Over a year ago

If you already have bytes, and you want to interpret them as bytes, then you don't want, or need, to do anything. Anything you add will just slow things down and make things more confusing, so why d it? If, on the other hand, you want to interpret them as X-byte big-endian integers, then you want to use int.from_bytes. Or, if you want to interpret them as C structs full of shorts and long, use struct.unpack_from. And so on. The point is the same: write the thing that does what you actually want in the most obvious way.

Collectives™ on Stack Overflow

Different ways to convert a byte into integer in python

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related