Bypass "Array is too big" python error

Question

I have a csv file which contains more than 200 000 lines of meteo data. When I want to model the data with matplotlib, this error appears:

Traceback (most recent call last):
  File "try4.py", line 19, in <module>
    X,Y = meshgrid( data_x,data_y )   
  File "C:\Python27\lib\site-packages\numpy\lib\function_base.py", line 3378, in  meshgrid
    mult_fact = np.ones(shape, dtype=int)   
  File "C:\Python27\lib\site-packages\numpy\core\numeric.py", line 148, in ones
    a = empty(shape, dtype, order) 
  ValueError: array is too big.

I found out that a file with 5000 lines max can be processed. How can I bypass the error in order to process all the file of 200000 lines? Here is my code :

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.mlab as mlab
from pylab import *


# read CSV as a numpy array
data = mlab.csv2rec('datasets/mix.csv')

# print CSV file headers
print data.dtype.names

# load columns as vectors
data_x = data['longitude']
data_y = data['latitude']
data_u = data['x']
data_v = data['y']

X,Y = meshgrid( data_x,data_y )
U = cos(data_u)
V = sin(data_v)


# plot raw data
Q = quiver( X, Y, U, V, units='width')
qk = quiverkey(Q, 0.5, 0.92, 2, '.', labelpos='W',
               fontproperties={'weight': 'bold'})
title('Current Surface')

plt.show()

What does an array element look like? This answer: stackoverflow.com/questions/13652650/… seems to think that there isn't an obvious array limit (and if so at least at 1m+ elements) — ydaetskcoR
– ydaetskcoR, Commented Jun 26, 2013 at 8:14
@ydaetskcoR Here is an example //headers X, Y, Latitude, Longitude, VOGRD_201304100000 //data 292, 1, 0.000000, -50.530000, 0 ...and 200 000 other lines like that — So4ne
– So4ne, Commented Jun 26, 2013 at 8:41
Can you give more information? Which matplotlib calls are you using, etc. — tiago
– tiago, Commented Jun 26, 2013 at 8:47
@tiago I import numpy, pyplot, mlab and pyplat. I want to plot current surface's data so I'm catching x, y, latitude and longitude and I manipulate it with quiver's function. — So4ne
– So4ne, Commented Jun 26, 2013 at 8:54
Please show us some code. We're not sure what you're doing that causes this. — brice
– brice, Commented Jun 26, 2013 at 9:04

tacaswell · Accepted Answer · 2013-06-26 09:47:46Z

2

why are you using meshgrid (doc)? It well generate a 200k by 200k array which will not match the dimensions of your u and v data. I think you want to do this

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.mlab as mlab
from pylab import *


# read CSV as a numpy array
data = mlab.csv2rec('datasets/mix.csv')

# print CSV file headers
print data.dtype.names

# load columns as vectors
data_x = data['longitude']
data_y = data['latitude']
data_u = data['x']
data_v = data['y']

U = cos(data_u)
V = sin(data_v)


# plot raw data
Q = quiver(data_x, data_y, U, V, units='width')
qk = quiverkey(Q, 0.5, 0.92, 2, '.', labelpos='W',
               fontproperties={'weight': 'bold'})
title('Current Surface')

answered Jun 26, 2013 at 9:47

tacaswell

88.2k23 gold badges222 silver badges203 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Bypass "Array is too big" python error

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related