Integer limit crossed while performing bitwise left shift operation in array

Question

I am creating a numpy array using the bitwise left shift operator.

For example, I create array p, where the shape of array is same as that of matrix a i.e. (23,):

>>> import numpy
>>> a = numpy.array([0,0,0,0,0,0,1,0,1,0,1,1,1,0,0,0,0,0,1,0,0,1,1])

>>> p = 1 << arange(a.shape[-1] - 1) #left shift

And the result is as expected:

>>> p
array([      1,       2,       4,       8,      16,      32,      64,
       128,     256,     512,    1024,    2048,    4096,    8192,
     16384,   32768,   65536,  131072,  262144,  524288, 1048576,
   2097152])

But, if we increase the size of array, let's say to (70,):

>>> a = numpy.array([0,0,0,0,0,0,1,0,1,0,1,1,1,0,0,0,0,0,1,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0])
>>> p = 1 << arange(a.shape[-1] - 1, -1, -1) #left shift 
>>> p
array([                   1,                    2,                    4,
                      8,                   16,                   32,
                     64,                  128,                  256,
                    512,                 1024,                 2048,
                   4096,                 8192,                16384,
                  32768,                65536,               131072,
                 262144,               524288,              1048576,
                2097152,              4194304,              8388608,
               16777216,             33554432,             67108864,
              134217728,            268435456,            536870912,
             1073741824,           2147483648,           4294967296,
             8589934592,          17179869184,          34359738368,
            68719476736,         137438953472,         274877906944,
           549755813888,        1099511627776,        2199023255552,
          4398046511104,        8796093022208,       17592186044416,
         35184372088832,       70368744177664,      140737488355328,
        281474976710656,      562949953421312,     1125899906842624,
       2251799813685248,     4503599627370496,     9007199254740992,
      18014398509481984,    36028797018963968,    72057594037927936,
     144115188075855872,   288230376151711744,   576460752303423488,
    1152921504606846976,  2305843009213693952,  4611686018427387904,
   -9223372036854775808,                    0,                    0,
                      0,                    0,                   16])

At the top you can see that as it increases from 1,2,4,8,..... becomes negative and then 0 and then 16 eventually.

This is not the case if I do it separately :

>>> 1<<70
1180591620717411303424

So, what can I do to make the elements of my array to have values corresponding to 1<<x, where x is a high number (greater than 70)?

Specifically python ints are variable precision, and are very heavy objects to support that. Numpy ints reflect fixed-width ints. — Max
– Max, Commented Sep 12, 2018 at 18:23

Benjamin · Accepted Answer · 2018-09-12 23:38:18Z

4

Python int and numpy int are not the same... Python supports arbitrary length, whereas numpy is fixed by the type:

numpy.array([1]) << 70
>>> array([64], dtype=int32)

One solution is to use the object dtype:

numpy.array([1], dtype=numpy.object) << 70
>>> array([1180591620717411303424], dtype=object)

And the following will work as expected:

a = numpy.array([1], dtype=numpy.object) << numpy.arange(70)

Looking at the type of the last element, we see that it is a Python int:

type(a[-1])
>>> <class 'int'>

edited Sep 12, 2018 at 23:38

answered Sep 12, 2018 at 18:24

Benjamin

12k13 gold badges75 silver badges121 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

appsdownload Over a year ago

Thank you. Your work around is awesome! But as you mentioned that object dtype has a great performance penalty. Can we do something to improve this? I am basically doing this to improve my performance!

Benjamin Over a year ago

@appsdownload: you should ask a separate question about the performance aspect...

Collectives™ on Stack Overflow

Integer limit crossed while performing bitwise left shift operation in array

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related