Why is this numpy array operation so slow?

Question

I am a python beginner and I am trying to average two NumPy 2D arrays with shape of (1024,1024). Doing it like this is quite fast:

newImage = (image1 + image2) / 2

But now the images have a "mask" that invalidate certain elements if set to zero. That means if one of the elements is zero, the resulting element should also be zero. My trivial solution is:

newImage = numpy.zeros( (1024,1024) , dtype=numpy.int16 )

for y in xrange(newImage.shape[0]):
   for x in xrange(newImage.shape[1]):
      val1 = image1[y][x]  
      val2 = image2[y][x]                            
      if val1!=0 and val2!=0:               
         newImage[y][x] = (val1 + val2) / 2

But this is really slow. I did not time it, but it seems to be slower by a factor of 100.

I also tried using a lambda operator and "map", but this does not return a NumPy array.

eumiro · Accepted Answer · 2011-04-06 14:20:29Z

8

Try this:

newImage = numpy.where(np.logical_and(image1, image2), (image1 + image2) / 2, 0)

Where none of image1 and image2 equals zero, take their mean, otherwise zero.

edited Apr 6, 2011 at 14:20

answered Apr 6, 2011 at 14:11

eumiro

214k36 gold badges307 silver badges264 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Tom Zych · Accepted Answer · 2011-04-06 13:59:48Z

2

Looping with native Python code is generally much slower than using built-in tools that use fast C loops. I'm not familiar with NumPy; can you use map() to do a transformation from your two input arrays to the output? If so, that should be faster.

answered Apr 6, 2011 at 13:59

Tom Zych

13.7k9 gold badges38 silver badges55 bronze badges

Comments

pberkes · Accepted Answer · 2011-04-06 14:18:55Z

1

Explicit for loops are very inefficient in Python in general, not only for numpy operations. Fortunately, there are faster ways to solve our problem. If memory is not an issue, this solution is quite good:

import numpy as np
new_image = np.zeros((1024, 1024), dtype=np.int16)
valid = (image1!=0) & (image2!=0)
new_image[valid] = (image1+image2)[valid]

Another solution using masked arrays, which do not create copies of the arrays (they represent views of the original image1/2:

m1 = np.ma.masked_equal(image1, 0)
m2 = np.ma.masked_equal(image2, 0)
new_image = (m1+m2).filled(0)

Update: The first solution seems to be 3 times faster than the second for arrays with about 1000 non-zero entries.

edited Apr 6, 2011 at 14:18

answered Apr 6, 2011 at 14:00

pberkes

5,3801 gold badge28 silver badges23 bronze badges

Comments

Arijan · Accepted Answer · 2014-07-08 22:08:13Z

0

numpy array access operation seems slow at best. I can't see any reason for it. You can clearly see it by constructing a simple example:

    import numpy
    # numpy version
    def at(s,n):
      t1=time.time()
      a=numpy.zeros(s,dtype=numpy.int32)
      for i in range(n):
        a[i%s]=n
      t2=time.time()
      return t2-t1
    # native version
    def an(s,n):
      t1=time.time()
      a=[(i) for i in range(s)]
      for i in range(n):
        a[i%s]=n
      t2=time.time()
      return t2-t1

    # test
    [at(100000,1000000),an(100000,1000000)]

Result: [0.21972250938415527, 0.15950298309326172]

answered Jul 8, 2014 at 22:08

Arijan

1731 silver badge6 bronze badges

Collectives™ on Stack Overflow

Why is this numpy array operation so slow?

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related