2

I have a large list of lists that I'm converting to a numpy array, and then iterating through, something like this:

a = [ [0,1,2], [3,4,5], [6,7,8] ]
np_a = np.array(a)

for i in np_a:
    print i # prints [0,1,2], then [3,4,5], then [6,7,8] as expected
    # do a bunch of stuff to each row

Of course this works, but since I'm dealing with a large body of data, this can take up to a minute to run. I've been looking into various ways to speed up this code, with the number one recommendation being to vectorize the operations I'm performing within the loop. However, the operations are not trivial to vectorize (think a bunch of dot products and transforms and such on each row...). One method I've found is to use the np.nditer method, like this:

a = [ [0,1,2], [3,4,5], [6,7,8] ]
np_a = np.array(a)

for i in np.nditer(np_a):
    print i # prints 0, then 1, then 2, etc...
    # do a bunch of stuff to each rowon 

But this seems to flatten the array into a 1-D list. I've looked through the documentation pretty extensively, and haven't found what I'm looking for here: https://docs.scipy.org/doc/numpy-1.13.0/reference/arrays.nditer.html#iterating-as-a-specific-data-type

Does anyone have any specific recommendations on how to make this work as I expect with np.nditer, or some other implementation that's more efficient than the standard 'for r in a' method?

11
  • 2
    So, have you profiled it? Where's the bottleneck? Commented Dec 25, 2017 at 19:24
  • 1
    If you're looping over an array, you're doing it wrong... Commented Dec 25, 2017 at 19:24
  • 1
    Read that nditer tutorial all the way to the end. It may, on occasion, help with complex broadcasting, but its real value is as a stepping stone to writing your code in cython (or other direct C code). Commented Dec 25, 2017 at 19:38
  • 1
    Maybe you find interesting numba and use gpu. Commented Dec 25, 2017 at 19:51
  • 1
    Try numba's @jit to gain some speed-up Commented Dec 25, 2017 at 21:05

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.