Pearson correlation matrix in Python, ValueError

Question

I'm trying to get a matrix, (lat, lon) size, with the Pearson Coefficient value for every grid point, for

x : a 3D DataArray (time, lat, lon) (time size is 30)

y : a DataArray column vector with a 30 values series inside

So i would like to calculate the pearson coefficient for every (lat,lon) for a column vector of 30 elements for x.

I tried:

corrmap = xr.DataArray(x2)
for i in range(len(corrmap['lat']))
     for j in range(len(corrmap['lon']))
          corrmap[i, j], p_value = pearsonr(x[:, i, j], y)

but i get this error:

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

that i cannot perfectly understand in the meaning. Is my method uncorrect? Should i use another type of code to solve my problem?

Any help would be greatly appreciated.

What is y.shape? (I'm assuming that a DataArray object has a shape attribute.) — Warren Weckesser
– Warren Weckesser, Commented Apr 9, 2019 at 16:36
Is pearsonr imported from scipy.stats? If not, where is it defined? — Warren Weckesser
– Warren Weckesser, Commented Apr 9, 2019 at 16:38
sorry I forgot to say that y.shape is (30, 1), so a column vector. And yes I did, i correctly imported pearsonr from scipy.stats — kadau92
– kadau92, Commented Apr 9, 2019 at 17:18
"... y.shape is (30, 1), so a column vector." That's the problem. See @JulianGiles' answer. — Warren Weckesser
– Warren Weckesser, Commented Apr 9, 2019 at 20:31

JulianGiles · Accepted Answer · 2019-04-09 19:30:44Z

2

The problem is you are using y and it has two dimensions, pearsonr can't handle that. Specify y[:,0] and it works. That is:

corrmap = np.zeros(((len(corrmap['lat']), len(corrmap['lon'])))

for i in range(len(corrmap['lat'])):
     for j in range(len(corrmap['lon'])):
          corrmap[i, j], p_value = pearsonr(x[:, i, j], y[:,0])

Also I would just use a numpy array for the coefficients instead of an xarray, at least to get the values, then you can convert it to xarray.

answered Apr 9, 2019 at 19:30

JulianGiles

3482 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Konstantin Grigorov · Accepted Answer · 2019-04-09 17:06:41Z

0

Assuming that you are using scipy.stats.pearsonr. The arguments to PearsonR should be one-dimensional arrays.

So give a go to:

corrmap[i, j], p_value = pearsonr(x[:, i, j].ravel(), y)

What ravel() does is return contiguous flattened array, Numpy ravel()

Here is also a bit of context around the error that you are getting, evaluate array in boolean context. In other words, the extra dimension is probably causing some operations to be applied to an array instead of a scalar. This issue is of the same nature as yours: SO: PearsonR ValueError

answered Apr 9, 2019 at 17:06

Konstantin Grigorov

1,67816 silver badges21 bronze badges

1 Comment

kadau92 Over a year ago

Thank you very much for your answer. Yes i correctly imported pearsonr from scipy.stats. Anyway, making the correction you suggest (that i understand and approve) i get an AttributeError, because of the fact that "'DataArray object' has no attribute 'ravel'. Is it a mistake to inizialize corrmap at the beginning as a xr.DataArray with a (lat, lon) shape?

Bendy Latortue · Accepted Answer · 2021-02-13 00:59:09Z

0

You can calutate the Pearson correlation coefficient using:

import numpy
numpy.corrcoef(list1, list2)[0, 1]

answered Feb 13, 2021 at 0:59

Bendy Latortue

4015 silver badges6 bronze badges

Collectives™ on Stack Overflow

Pearson correlation matrix in Python, ValueError

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related