perl - array of integers using way too much memory?

All Perl values are represented internally as perl scalars, which consume way more memory than a simple int. Even if the scalar is only holding an int. Even if the scalar is undef!

As others have suggested, PDL may be something to look at if you really want to work with huge arrays of this sort.

You can always use C or C++ in Perl.This will probably give you a small footprint in some hard jobs. Just an idea using C!

#!/usr/bin/perl
use Inline C;
use strict;

for(my $i=0; $i<5000000; $i++) {
        set_array_index($i,$i);
        if($i % 1000000 == 0) {
                #print "$i\n";
                print get_array_index($i)."\n";
        }
}

__END__
__C__

int array[5000000];

void set_array_index(int index,int value) {
    array[index]=value;
}

int get_array_index(int index) {

    if (array[index]==NULL)
        return 0;

    return array[index];
}

Complete revision of my answer. Looking at what you have in your code, I see some strange things.

my @arr = [1..5000000];

Here, you assign an anonymous array-reference to $arr[0]. This array only holds one value: The array reference. The hidden anonymous array holds the 5 million numbers.

for($i=0; $i<5000000; $i++) {
        $arr[$i] = $i;
        if($i % 1000000 == 0) {
                print "$i\n";
        }
}

Here, you fill the array with 5 million sequential numbers, overwriting the array reference in the declaration.

A much shorter way to do it would be:

my @arr = (1 .. 5_000_000);

Perhaps that will save you some memory.

Here are a few lines from an interactive pdl2 session showing how this could be done using basic PDL constructs:

pdl> $arr = sequence(long, 5000000) + 1;  # create pdl data array (a.k.a. a piddle)

pdl> help vars                            # see, it is only ~19MB
PDL variables in package main::

Name         Type   Dimension       Flow  State          Mem
----------------------------------------------------------------
$arr           Long D [5000000]            P           19.07MB
$Pi          Double D []                   P            0.01KB

pdl> p which( $arr%1000000 == 0 )         # which returns indexes which are true
[999999 1999999 2999999 3999999 4999999]

See the on-line PDL book for a good introduction to what PDL can be used for. The PDL mailing lists are the best source of information on PDL usage and development. The responses are often rapid.

Rather than creating an array, you may create a binary string that is exactly 5000000 * 4 characters long using pack:

my $numbers = "";
$numbers .= pack("N", $_) for (1..5000000);

It should definitely take less space. Consequently, you can fetch the values with substr and unpack.

Maybe you could use an iterator instead of such a big list of integers.

The iterator pays an overhead of a function call for each new value, but saves memory. Check MJD Higher Order Perl Chapter 4 ( 4.2.1 ).

If I remember it right, range operator dont' build such a huge list in latest perls.

Or, implicitly handling the pack for you, there is Tie::Array::PackedC:

use Tie::Array::PackedC;
# make @arr use $arr_storage for storing packed elements, by default using 'l!' pack format
tie my @arr, 'Tie::Array::PackedC', my $arr_storage;

vec may also be of interest.