python bit shifting with numpy

Question

I'm working with 64 bit unsigned integers and after bit shifting comparing the value before decoding the rest of the bit values. I'm iterating over millions of values and trying to minimize process time.

The issue is bit shifting is not supported with uint64 nor numpy-uint64. I'm trying to avoid using int64 to avoid negative values.

example data: 0x8204000000000080 after shifting(word>> 60): =-8 #but comparing to 0x8

looping one million times and seeing how long it takes it was found that of all methods the '>>' shift operator was the most expedient with the next best option to call the abs() function. Is there a better more expedient solution for this?

Loop code:

import numpy as np
import time

start_time= time.time()
for i in range(1000000):
    x= np.int64(-1)
    x=np.right_shift(x,60)
print (time.time()-start_time)

start_time= time.time()
for i in range(1000000):
    x= np.uint64(-1)
    x=int(x/(2**60))
print (time.time()-start_time)

start_time= time.time()
for i in range(1000000):
    x= np.int64(-1)
    x=abs(x>>60)
print (time.time()-start_time)

start_time= time.time()
for i in range(1000000):
    x= np.int64(-1)
    x= x>>60
print (time.time()-start_time)

Output:

2.055999994277954
3.1540000438690186
0.619999885559082
0.5810000896453857

What makes you say shifting isn't supported? Because it totally is. — user2357112
– user2357112, Commented May 28, 2015 at 18:02

user2357112 · Accepted Answer · 2018-06-01 18:07:57Z

19

The issue is that when you apply the shift to an array scalar, NumPy tries to produce an output type that can hold all values of both input dtypes (with a Python int cast to either int32 or int64). There is no integer dtype that can hold all values of both uint64 and a signed dtype, and floats aren't an option here.

When one operand is an array and the other is a scalar (here a Python int), NumPy attempts to stuff the scalar into a smaller dtype, which for most shift operations means the shift amount is cast to int8 or uint8, depending on whether the other operand is signed. uint64 and uint8 both fit in uint64.

You'll have to cast the shift amount to an unsigned int:

>>> numpy.uint64(-1) >> 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: ufunc 'right_shift' not supported for the input types, and the inputs
 could not be safely coerced to any supported types according to the casting rul
e ''safe''
>>> numpy.uint64(-1) >> numpy.uint64(1)
9223372036854775807

edited Jun 1, 2018 at 18:07

answered May 28, 2015 at 18:15

user2357112

286k32 gold badges491 silver badges571 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Joran Beasley Over a year ago

ahh good stuff ... not sure why it worked fine for me in older python and older numpy :P (+1 though since you could reproduce original issue)

kaminsknator Over a year ago

Is this the same condition then for all operations with the array scalar? for bitwise & it would be necessary to: data_word & np.uint64(mask) Thanks for your feedback it answers my question.

Joran Beasley · Accepted Answer · 2015-05-28 18:10:08Z

2

>>> import numpy
>>> a = numpy.array([1,2,3],dtype=numpy.uint64)
>>> a>>2
array([0, 0, 0], dtype=uint64)

>>> a = numpy.array([1,2,2**64-1],dtype=numpy.uint64)
>>> a>>2 
array([0, 0, 4611686018427387903], dtype=uint64)
>>> a>>60
array([ 0,  0, 15], dtype=uint64)

I dont understand the problem perhaps?

edited May 28, 2015 at 18:10

answered May 28, 2015 at 18:00

Joran Beasley

114k13 gold badges168 silver badges187 bronze badges

8 Comments

dlask Over a year ago

Perhaps a typo. The uint64 type seems to be preserved in the response.

kaminsknator Over a year ago

in the output it looks like it converted your data type to int64. My data can't be converted to int64 as it would lose information. The largest possible positive number with int64 is 2^63-1 while the largest possible positive number with uint64 is 2^64 I believe. My data contains numbers in this range.

Joran Beasley Over a year ago

it didnt it was a typo its still uint64 (I initially used int64 because I misread the original statement and didnt correct the output when I changed to uint)

kaminsknator Over a year ago

The error: TypeError: ufunc 'right_shift' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'' type:<class 'numpy.uint64'> operation that causes the error: (data_word>>60)

Joran Beasley Over a year ago

hmmm what version of python are you using? it seems fine to me in python2.6 and numpy 1.9.0

|

Collectives™ on Stack Overflow

python bit shifting with numpy

2 Answers 2

2 Comments

8 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

8 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related