Python lambda application ambiguity on multi-dimensional arrays

Question

Given the code example below, one produces an expected result and the other gives an error. Seems confusing for a beginner (i.e. me). I assume the arithmetic operations work element wise but others don't. What's a "good" (i.e. efficient) generalize way to simply perform operations on elements of a multi-dimensional array without having some underlying knowledge of the array behavior?

import numpy as np

data = np.array([[1, 2, 3, 4], [5, 6, 7, 8]])
print(data)

my_function = lambda x: x*2+5

result = my_function(data)
print(result)

Output: [[1 2 3 4] [5 6 7 8]] [[ 7 9 11 13] [15 17 19 21]]

import numpy as np

data = np.array([[1, 2, 3, 4], [5, 6, 7, 8]])
print(data)

my_function = lambda x: x if x < 3 else 0

result = my_function(data)
print(result)

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Edit: I am not looking for a particular solution. Yes, I can use np.where or some other mechanisms for this exact example. I am asking about lambdas in particular and how their use seems ambiguous to the user. If it helps, the lamba / filter is coming from command line/outside of module. So it can be anything the user wants to transform the original array to - easy as square elements, or call an API and then use its output to determine the replacement value. You get the idea.

Running python 3.9.13

Please clarify the question but it's a multi-dupe anyway. np.where(data < 3, data, 0). — Michael Szczesny
– Michael Szczesny, Commented Jun 16, 2022 at 17:59
In any case, this seems to have been addressed in the previous comment, there is no generic way to apply a function to every element in a numpy.ndarray in a way that is efficient and vectorized. You can use the numpy.vectorize factory function, but that is essentially a for-loop under the hood, and is meant for convenience not performance — juanpa.arrivillaga
– juanpa.arrivillaga, Commented Jun 16, 2022 at 18:15
np.vectorize(my_function)(data)? If you don't care about performance. — Michael Szczesny
– Michael Szczesny, Commented Jun 16, 2022 at 18:15

hpaulj · Accepted Answer · 2022-06-16 18:28:45Z

3

This works because operators like * and + work element-wise for numpy arrays:

In [101]: data = np.array([[1, 2, 3, 4], [5, 6, 7, 8]])
     ...: print(data)
     ...: 
     ...: my_function = lambda x: x*2+5
     ...: 
     ...: result = my_function(data)
[[1 2 3 4]
 [5 6 7 8]]

my_function = lambda x: x if x < 3 else 0 fails because if x<3 is inherently a scalar operation. if/else does not iterate; it expects a simple True/False value

In [103]: data<3
Out[103]: 
array([[ True,  True, False, False],
       [False, False, False, False]])

np.vectorize is the most general tool for applying an array (or arrays) element-wise to a scalar function:

In [104]: f = np.vectorize(my_function, otypes=[int])

In [105]: f(data)
Out[105]: 
array([[1, 2, 0, 0],
       [0, 0, 0, 0]])

I included the otypes parameter to avoid one of the more common vectorize faults that SO ask about.

np.vectorize is slower than plain iteration for small cases, but becomes competative with large ones. But it's main advantage is that it's simpler to use for multidimensional arrays. It's even better when the function takes several inputs, and you want to take advantage of broadcasting.

answered Jun 16, 2022 at 18:28

hpaulj

233k14 gold badges260 silver badges392 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Heems Over a year ago

Thank you for the explanation. Aside from the toy data example above, I am actually dealing with rather large 2D arrays - for example shape = (16800, 43200). The vectorize method takes a very long time, but simply doing np.where(data < 3, data, 0) is wicked quick on these large array. What is np.where doing that's so much more efficient for this non-arithmetic operation?

hpaulj Over a year ago

Are you familiar with masked indexing? arr[mask]= a, arr[~mask]=b?

Collectives™ on Stack Overflow

Python lambda application ambiguity on multi-dimensional arrays

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related