Unexpected memory error when regriding data with scipy interpolate griddata method

Question

I have a 3000x6000 2D grid (from a tiff image). I want to regrid it into a lower resolution grid using griddata method from scipy.interpolate library. First, I need to form a 18000000x2 numpy array as the input for griddata based on what I read here. Here's what I do:

import numpy as np
from scipy.interpolate import griddata
x_length = 6000
y_length = 3000
def func(x, y):
    return x*(1-x)*np.cos(4*np.pi*x) * np.sin(4*np.pi*y**2)**2
grid_x, grid_y = np.meshgrid(np.linspace(0,1,x_length),np.linspace(0,1,y_length))
points = np.random.rand(x_length*y_length, 2)
values = func(points[:,0], points[:,1])
grid_z0 = griddata(points, values, (grid_x, grid_y), method='nearest')

I get a MemoryError when doing griddata. I have 8 gb of RAM and I shouldn't get this error based on the first answer to this question.

Overall, regriding a 3000x6000 grid into a lower resolution grid shouldn't be that hard and I guess I am doing something funny here. Should I get e MemoryError doing these liens of codes with 8 gb RAM?

P.S: Although I have a 64-bit operating system (Windows 7), I use the following Python version:

Python 2.7.2 (default, Jun 12 2011, 15:08:59) [MSC v.1500 32 bit (Intel)] on win32

It works for me. I have 4 GB of RAM and more than 3 are already occupied. In fact, in htop I only see around 200 MB, and each one of your arrays is exactly 288 MB. — Davidmh
– Davidmh, Commented Aug 6, 2014 at 23:13
Also, use x = np.round(x, 5) to round to the first 5 decimals. — Davidmh
– Davidmh, Commented Aug 6, 2014 at 23:15
@Davidmh I edited the question a fair amount. I am still getting a MemoryError running the code. Do you get an error? — ahoosh
– ahoosh, Commented Aug 7, 2014 at 0:31
This should give you the size of all the arrays accesible by Python: 1e-6 * sum(x.nbytes for x in globals().values() if isinstance(x, np.ndarray)). As expected, I get the same result as before, 720 MB. — Davidmh
– Davidmh, Commented Aug 7, 2014 at 0:50

DrV · Accepted Answer · 2014-08-07 09:54:47Z

3

As the comments point out, you run out of memory. A 32-bit Python running on 64-bit Windows 7 is limited to 2 GB of memory, and you just banged into that.

There are three solutions:

Get a 64-bit Python. (suggested)
Interpolate in several chunks (split the image into some suitable overlapping parts) (laborious)
Rethink your interpolation method (recommended)

If you have a regular grid (as in the case of an image), using griddata to regrid it into another regular grid is quite wasteful in terms of memory and time.

There are several methods to downsample the image. At least PIL and cv2 modules offer downsampling functions. If you want to use a SciPy method, have a look at scipy.ndimage.zoom. It will allow you to resample the image from one regular grid to another.

answered Aug 7, 2014 at 9:54

DrV

23.7k8 gold badges66 silver badges75 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

ahoosh Over a year ago

I will definitely change my Python to 64-bit. It was useful to know I am only using 2 GB of my RAM! If I already have a 2D numpy array (2D grid) and don't have access to the tiff file anymore, what is the most efficient way to do it? Do you think using RegularGridInterpolator from scipy.interpolate would be a good option?

DrV Over a year ago

If you have the image data in a 2D array, then the easiest way is to use something like scipy.ndimage.zoom(my_img_array, 0.43) to scale both directions by 0.43 (or whatever you want).

Collectives™ on Stack Overflow

Unexpected memory error when regriding data with scipy interpolate griddata method

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related