How much memory in numpy array? Is RAM a limiting factor?

Question

I'm using numpy to create a cube array with sides of length 100, thus containing 1 million entries total. For each of the million entries, I am inserting a 100x100 matrix whose entries are comprised of randomly generated numbers. I am using the following code to do so:

import random
from numpy import *

cube = arange(1000000).reshape(100,100,100)

for element in cube.flat:
    matrix = arange(10000).reshape(100,100)
    for entry in matrix.flat:
        entry = random.random()*100
    element = matrix

I was expecting this to take a while, but with 10 billion random numbers being generated, I'm not sure my computer can even handle it. How much memory would such an array take up? Would RAM be a limiting factor, i.e. if my computer doesn't have enough RAM, could it fail to actually generate the array?

Also, if there is a more efficient to implement this code, I would appreciate tips :)

Assuming double precision, at 8 bytes each, if you really are trying to store 10 billion of them, that's 80GB. If you have to ask, your computer doesn't have enough memory. That said, it looks like you're creating them all but not storing them, so you should be fine. — Gabe
– Gabe, Commented Jun 28, 2012 at 21:56
Does this answer your question? Python memory usage of numpy arrays — YaOzI
– YaOzI, Commented Jan 13, 2023 at 4:36

David Wolever · Accepted Answer · 2012-06-28 22:07:02Z

23

A couple points:

The size in memory of numpy arrays is easy to calculate. It's simply the number of elements times the data size, plus a small constant overhead. For example, if your cube.dtype is int64, and it has 1,000,000 elements, it will require 1000000 * 64 / 8 = 8,000,000 bytes (8Mb).
However, as @Gabe notes, 100 * 100 * 1,000,000 doubles will require about 80 Gb.
This will not cause anything to "break", per-se, but operations will be ridiculously slow because of all the swapping your computer will need to do.
Your loops will not do what you expect. Instead of replacing the element in cube, element = matrix will simply overwrite the element variable, leaving the cube unchanged. The same goes for the entry = random.rand() * 100.
Instead, see: http://docs.scipy.org/doc/numpy/reference/arrays.nditer.html#modifying-array-values

edited Jun 28, 2012 at 22:07

answered Jun 28, 2012 at 21:59

David Wolever

156k94 gold badges365 silver badges513 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

ali_m Over a year ago

Things will indeed 'break' if you exceed the total amount of available virtual memory, which seems very likely in this case since not many people have >80GB of swap space

Max Candocia Over a year ago

This is not entirely the case when dealing with functions such as np.zeros(). Lazy loading is used (at least in Linux versions), which will avoid using large amounts of memory until certain elements are accessed. For example, you can make a matrix with np.zeros((24000,24000)), and it does not take up much memory, but if you do np.random.random((24000,24000)), it takes up a bit over 4 GB. Better explanation: stackoverflow.com/questions/27574881/…

Phil Cooper · Accepted Answer · 2012-06-28 22:39:12Z

2

for the "inner" part of your function, look at the numpy.random module

import numpy as np
matrix = np.random.random((100,100))*100

answered Jun 28, 2012 at 22:39

Phil Cooper

5,8871 gold badge27 silver badges41 bronze badges

Collectives™ on Stack Overflow

How much memory in numpy array? Is RAM a limiting factor?

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related