Nested loops work too slow in Python

Question

Just figured out that 2 nested loops work extremely slow in Python. CPU load stays around 0% but it still works slow. Why? How can I fix that?

Initialization (shouldn't comment it to make it work fast):

    a = imresize(image, (maxY, maxX), 'lanczos')
    b = imresize(image, (maxY * 2, maxX), 'lanczos')

Slow code:

    result = np.empty((maxY, maxX, 3), dtype=np.uint16)
    for y in range(maxY):
        for x in range(maxX):
            result[y, x] = [a[y, x], a[y, x], a[y, x]]

And this one works even more slow:

    result = np.empty((maxY, maxX, 3), dtype=np.uint16)
    for y in range(maxY):
        for x in range(maxX):
            result[y, x] = [a[y, x], b[y*2, x], b[y*2+1, x]]

Is there any other more effective code to achieve the same results?

Shape of a is (299, 299), b - (598, 299), result - (299, 299, 3). I call the code about 5000 times (and wait about 10 minutes for that amount of data).

If I comment the provided code everything works just a second.

1) fix your algorithm, 2) use numpy 3) use CPython are all options. Without more info, it's hard to say anything — juanpa.arrivillaga
– juanpa.arrivillaga, Commented Sep 21, 2017 at 0:20
No info about any of those variables or your cpu-load evaluation is given... — sascha
– sascha, Commented Sep 21, 2017 at 0:21
Always, always, always use NumPy when your doing operations on every element of an array. — o11c
– o11c, Commented Sep 21, 2017 at 0:22
Show a more complete code (definition and shapes of all those vars) and tell us how you measured cpu-load. — sascha
– sascha, Commented Sep 21, 2017 at 0:23
You said you have arrays of uint8, but did not use numpy already. That does not make much sense (in most cases). — sascha
– sascha, Commented Sep 21, 2017 at 0:25

Sraw · Accepted Answer · 2017-09-21 01:12:52Z

2

Allocating memory is always slow in every language, so you need to avoid it. In your example, you create a list in every loop.

You should use something like the following:

result = np.empty((maxY, maxX, 3), dtype=np.uint16)
for y in range(maxY):
    for x in range(maxX):
        result[y, x, 0] = a[y, x]
        result[y, x, 1] = a[y, x]
        result[y, x, 2] = a[y, x]

Or as mentioned by @user3237718, you should use:

result = np.empty((maxY, maxX, 3), dtype=np.uint16)
for i in range(3):
    result[:, :, i] = a

The key point is to avoid dynamically allocating memory.

answered Sep 21, 2017 at 1:12

Sraw

20.6k11 gold badges61 silver badges93 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Dmitry Over a year ago

Already tried the first version of the code myself. It helped only a little.

Sraw Over a year ago

@Dmitry Try the second one, it helps a lot.

MIAWANG · Accepted Answer · 2017-09-21 01:01:38Z

1

please use vectorization to fix your code. like that:

result[:,:,0] = a

answered Sep 21, 2017 at 1:01

MIAWANG

1088 bronze badges

1 Comment

Dmitry Over a year ago

Nice idea! But I still have to use one loop with it.

Collectives™ on Stack Overflow

Nested loops work too slow in Python

2 Answers 2

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related