0

Is there a way to work with iterators instead of (for example) numpy.ndarray in numpy?

For example, imagine I have a 2D-array and I want to know if there is a row that only contain even numbers: 

import numpy as np

x = np.array([[1, 2], [2, 4], [3, 6]])
np.any(np.all(x % 2 == 0, axis=1))

Is there a way to do this kind of things without instantiating the intermediate objects in memory? (or maybe it is already the case and I just don't know it) In this example, that would mean having an iterator over [False True False] instead of an array. In other words, can we do something that would be equivalent to:

has_an_even_row = False 
for row in x:
    if np.all(row % 2 == 0):
        has_an_even_row = True
        break

My question doesn't only concern all and any but all function/methods in numpy. If it isn't possible I wonder if there is a practical reason for not having this in numpy. (Maybe everyone thinks it's useless, that would be a good reason)

Improve this question
7
  • 1
    Sure you can iterate over the rows. Use the usual python for loop. But be ware the action will usually, but not always, be slower. Commented Mar 24, 2019 at 18:03
  • I just updated my question, I'm looking for a solution internal to numpy. Commented Mar 24, 2019 at 18:04
  • What exactly are you envisioning? If the creation of intermediate objects is problematic, look into numexpr. But, as hpaulj is saying, if you want an iterator, use a for-loop. Commented Mar 24, 2019 at 18:05
  • 1
    You can also look at numba which is a JIT compiler that will just-in-time-compile functions that use simple loops over numpy data structures into native code. In my experience, it is quite effective. Commented Mar 24, 2019 at 18:17
  • 1
    numpy is like a Lego set. It is fast and easy to use when you stick with the given building blocks. It does not include a custom block molding machine - you have to get that from some other source. Commented Mar 24, 2019 at 20:08

2 Answers 2

Reset to default
1

The number of temporary arrays may be more than you realize:

In [224]: x = np.array([[1, 2], [2, 4], [3, 6]])                                
In [225]: x % 2                                                                 
Out[225]: 
array([[1, 0],
       [0, 0],
       [1, 0]])
In [226]: _ == 0                                                                
Out[226]: 
array([[False,  True],
       [ True,  True],
       [False,  True]])
In [227]: np.all(_, axis=1)                                                     
Out[227]: array([False,  True, False])
In [228]: np.any(_)                                                             
Out[228]: True

In this case, working row by row would save on calculating the last row's values.

The last any step might short-circuit, stopping when it hits the True - that's an implementation detail.

A thoroughly iterative, no excess calculations method would be something like:

In [231]: val = False 
     ...: for row in x: 
     ...:     for col in row: 
     ...:         if col%2!=0: 
     ...:             break 
     ...:         val=(row,col) 
     ...:         break 

In [232]: val                                                                   
Out[232]: (array([2, 4]), 2)

This approach would make sense if I were writing in C or a lisp like language, where testing, memory management, and calculations all occur at the same code level. But it wouldn't be very modular or reusable.

The idea underlying numpy is to provide a comprehensive set of compiled building blocks. Those blocks won't be optimal for all tasks, but on the whole they are fast and easy to use.

It's generally recommended to use the given building blocks for quick development. Once it's working then worry about improving the speed of time critical steps.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

"But it wouldn't be very modular or reusable" I agree, that's why I was wondering if it existed inside numpy. "It's generally recommended to use the given building blocks for quick development. Once it's working then worry about improving the speed of time critical steps.", basically if I had a use case where memory is limiting you would advice to rewriting the code in Cython or any lower level language?
1

The numpy library doesn't give you very many tools to use some of the conventional Python protocols because it is focused on performance within a narrow domain (numeric computation). The whole purpose of numpy is to do numeric operations that are slow in pure Python much more quickly (close to your hardware's maximum speed, like code written in a lower level language like C) without loosing all of the benefits of Python (like garbage collection and easy to read syntax).

The downside to focusing on a narrow domain is that you lose some benefits of more general code. So your for loop code can do less work than numpy does, because it can short-circuit, breaking out of the iteration as soon as the result is known. It doesn't need to do the modulus for every row if it found the result it needs already.

But I suspect if you test it, your numpy code may still going to be faster a lot of the time (test on real data, not trivial stuff like in your example)! Even though it computes a whole bunch of intermediate results up front, the low level operations are so much faster than the equivalent in pure Python that it doesn't matter that it has to iterate over the whole array.

Improve this answer

6 Comments

I'll surely try to compare time performances and come back here :). But that wouldn't really be a sufficient reason for not having a way to have iterators, there probably is a memory-speed tradeoff here. Unless I'm missing something.
Well, I guess I just don't understand exactly what you're expecting. Numpy arrays are iterable, so you can write normal Python code to operate on them (though it may not be as convenient or even as fast as using normal Python data structures). Many numpy functions only work on arrays, rather than iterables, and the reason for that is that their performance benefits are only available for arrays, not for arbitrary objects.
"and the reason for that is that their performance benefits are only available for arrays" that's the part I really don't understand, from what I understand numpy takes advantage of static typing (together with type homogeneous structures) to speedup things and save memory. What I fail to understand is why this can't be used to build some other (statically typed) set of functions that instead of having arrays as both input and output would have arrays as input and iterators as output (a custom kind of iterator since it wouldn't iterate over arbitrary object, but instead over a given type).
Testing this is a bit hard as it requires re-writing some parts of numpy, but I'll try to in the near future if nobody tells me it's just not possible because of some reason I fail to see for now (maybe because I have an over-simplified vision of how numpy works).
The iterator protocol isn't that specific. You can't really have a function that only accepts one kind of iterator and say that's using the iterator protocol. You either call next on the arbitrary iterator object you've been given (which is slow, since it does a Python function call and might run arbitrary Python code), or you don't.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.