C Storing Huge Arrays

Question

Thanks for taking the time to read this. I've been learning C for a few days and am stuck. I'm playing around with creating huge arrays (several GB) and cannot seem to create an array that's bigger than 2GB. Here is my code:

#include <stdio.h>                                                              
#include <stdlib.h>                                                             
#include <math.h>                                                               

/* Exploring 1d array sizes in c */                                             

int main()                                                                      
{                                                                               
    int i;       
    double mbs;                                                                 
    int arr_length = 260000000;                                                 
    int arr_size = sizeof(double) * arr_length;                                 
    double *arr = malloc(arr_size);                                             

    for(i = 0; i < arr_length; i++)                                             
        arr[i] = (double)i;                                                     

    /* Print array size */                                                      
    arr_size = (double)arr_size;                                                
    mbs = (double)arr_size / pow(1024.0, 2);                                    
    printf("The size of the array is %1.1f megabytes. \n", mbs);                

    return 0;                                                                     
}

When I run the code, I get a reasonable result:

:~/c-examples> gcc -o array-size array-size2.c
:~/c-examples> ./array-size 
The size of the array is 1983.6 megabytes.

However, if I increase arr_length to 270000000 (270 million), I get a segmentation fault even though the array would be just over 2GB in size. I'm currently running 64 bit OpenSuse 13.1, and have 6GB of RAM:

:~/c-examples> free -h
             total       used       free     shared    buffers     cached
Mem:          5.6G       910M       4.7G        27M        12M       377M
-/+ buffers/cache:       520M       5.1G
Swap:         2.0G       307M       1.7G

I was hoping to eventually be able to store arrays of size 10-12GB (after adding more RAM), but want to make sure I understand exactly what's going on before. Thanks again for your time - and suggestions (an criticism!) is most welcome.

Are you sure your GCC is 64-bit, too? A 64-bit OS doesn't necessarily equate to a 64-bit compiler, as my 64-bit Win7 setup with 32-bit MinGW GCC can testify. If your GCC is creating a 32-bit build, 32-bit restrictions will apply. — cf-
– cf-, Commented Mar 31, 2014 at 3:55
this really feels like an issue linked to compiling on 32 bits, as 2**32 == 4.3G and if you're using an int somewhere instead of a uint, it will be maxed exactly around 2.15G — Jules Gagnon-Marchand
– Jules Gagnon-Marchand, Commented Mar 31, 2014 at 3:58
The problem is that you are overflowing arr_size beyond the maximum 32-bit positive integer value when you change arr_length to 270000000, and you end up passing a negative size to malloc, which will then return NULL. If you change the declaration of arr_size from an 'int' to 'size_t', you will then see that you can successfully allocate the memory you are asking for. — mkimball
– mkimball, Commented Mar 31, 2014 at 4:00
Thanks for all your comments, arr_size was indeed incorrectly defined as an int, I feel quite dumb for missing that. Thanks again! — MAB
– MAB, Commented Mar 31, 2014 at 4:09
BTW: When code prints size_t, use size_t sz; printf("%zu\n", sz); — chux
– chux, Commented Mar 31, 2014 at 4:15

Maxim Razin · Accepted Answer · 2014-03-31 03:57:18Z

5

int is 32 bit on x86 linux, even in 64 bit mode. It means that int cannot hold values larger than 2^31-1 (i.e. 2G)

Try to change the type of arr_length and arr_size to size_t

answered Mar 31, 2014 at 3:57

Maxim Razin

9,5067 gold badges36 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

MAB Over a year ago

Thanks! Turns out it arr_size was overflowing by just a bit - I feel stupid for not realizing that. I've changed the variable types to size_t and everything works great. Thanks a lot!!

Collectives™ on Stack Overflow

C Storing Huge Arrays

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related