numpy array able to allocate insane amounts of memory for large arrays

Question

I am new to numpy. This might seem a bit stupid to most people but I couldn't find the answer anywhere

x = np.zeros(1000000000000)
x.nbytes
>> 8000000000000

this is like 8000 GB of memory. Initially i thought nbytes is not the actual bytes of memory consumed, but its written clearly here that nbytes gives the total bytes.

How is this possible? Does numpy use something like a lazy load? I am sure something is wrong here as 8000 GB of memory is simply impossible in my machine which has 8 GB of RAM.

PS: in google colab this leaks memory

bhuvy · Accepted Answer · 2018-11-24 23:39:44Z

This probably has something to do at the system level. I don't know how google's systems exactly are, but even on a standard linux machine with 8GB of memory this would be possible because as you said lazy loading of pages.

numpy would probably use the stdlib malloc on my machine, which for large allocations uses the system mmap. The system has to zero out pages given to a process, or it runs a security risk, meaning newly mmap'ed pages are zero'ed by default. Since you haven't used any of the pages, the system doesn't allocate any for you.

#include <stdio.h>
#include <unistd.h>
#include <sys/mman.h>

int main() {
    void *ptr = mmap(0, 8000000000000, PROT_READ | PROT_WRITE, MAP_SHARED, 0, 0);
    if (ptr == NULL) {
            printf("Allocating bytes failed\n");
    }
    else {
            printf("Success!\n");
    }
}

My hunch is you are in for a surprised (lots of thrashing) if you try to use all of your memory at once though :D.

Collectives™ on Stack Overflow

numpy array able to allocate insane amounts of memory for large arrays

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related