Count non-nan-values in 3d numpy array

Question

I have a list of N 2d numpy arrays, all of the same size Mx3, all of which represent a single sample of M coordinates. Sometimes the value of a coordinate can be np.nan.

I (think I) know how to compute the average and standard deviation over these coordinate samples, namely as follows (i.e. stack them and compute the average and std along axis=0):

averaged = np.nanmean(np.array(listOf2dArrays, dtype=np.float64), axis=0)

std      = np.nanstd( np.array(listOf2dArrays, dtype=np.float64), axis=0)

How can I determine the count on which the average and std values for each coordinate are based i.e. the number of non-nan values for each coordinate(component) m?

The result should be a 2d-array of dimensions Mx3 containing non-nan-count values.

Could you include an example list of arrays that need to be processed, along with the expected results? — The_spider
– The_spider, Commented Aug 15, 2022 at 11:10
np.count_nonzero(~np.isnan(np.array(listOf2dArrays, dtype=np.float64)), axis=0) — Stef
– Stef, Commented Aug 15, 2022 at 11:26
@Stef Thanks, got to the same conclusion after thinking a bit more along the lines of the avg and std code. Nice one-liner.Any advice on processing this result further to a 1d array containing the counts when they are all equal along a row? I.e. which contains the value in case of equality accross a row or np.nan otherwise. I currently solved this using a for-loop accross the rows and check using np.all(count_xyz[i]==count_xyz[i][0]) — TheCat
– TheCat, Commented Aug 15, 2022 at 12:33

msi_gerva · Accepted Answer · 2022-08-15 11:26:22Z

You can make a mask array and just sum 0s and 1s, where 0 either means a real value or nan value.

Basically, let us assume you have a 3D array with some random nan-values and let us count the number of nans along some axis (in your case the axis 0):

#!/usr/bin/env ipython
# ----------------------------
import numpy as np

nx,ny,nz=50,50,50;npts=nx*ny*nz

n_nans=100
# ----------------------------
# Let us make a random array with number of NaN values n_nans 
A=np.random.random((nz,ny,nz))
i_nan=np.array(np.random.random((n_nans))*npts,dtype='int32')
dum = A.flatten();
dum[i_nan] = np.nan;
A=np.reshape(dum,np.shape(A))
# ----------------------------

A_nanmask=np.zeros(np.shape(A));A_nanmask[np.isnan(A)]=1;
A_notnanmask=np.ones(np.shape(A));A_notnanmask[np.isnan(A)]=0;
# -----------------------------
# Get some count of nans:
A_notnan = np.sum(A_notnanmask,axis=0)
A_nan = np.sum(A_nanmask,axis=0)

# Check the sums:
A_total = A_notnan+A_nan

So, the A_total should be 50 in this example.

Stef · Accepted Answer · 2022-08-16 13:50:33Z

0

You can do it the same way using count_nonzero (which is preferred over sum):

np.count_nonzero(~np.isnan(np.array(listOf2dArrays, dtype=np.float64)), axis=0)

answered Aug 16, 2022 at 13:50

Stef

30.9k3 gold badges34 silver badges60 bronze badges

Collectives™ on Stack Overflow

Count non-nan-values in 3d numpy array

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related