Need advice for Memory usage in perl

Question

I am not new to perl , but I couldn't able to solve this usecase.

Here is my problem statement:

I am using Activestate perl 5.12 32-bit. I want to hold some 32 x 8MB array of double in Memory at a time. unfortunately, I am getting 'out of memory' error in this case. for example, the code below will lead to out of memory.

my $aref1 = [(.25) x (8*1024*1024)];
my $aref2 = [(.25) x (8*1024*1024)];
my $aref3 = [(.25) x (8*1024*1024)];
my $aref4 = [(.25) x (8*1024*1024)];
my $aref5 = [(.25) x (8*1024*1024)];
my $aref6 = [(.25) x (8*1024*1024)];
my $aref7 = [(.25) x (8*1024*1024)];
my $aref8 = [(.25) x (8*1024*1024)];
my $aref9 = [(.25) x (8*1024*1024)];
my $aref10 = [(.25) x (8*1024*1024)];

Is there any way to efficiently handle it?

Note: In any case, the access to the arrays is needed at any time of execution as fast as possible (since the user cannot wait soo long)

the options I have tried:

DBM::Deep - It takes more time
PDL - Gives 'out of memory' error for 32 x 8MB array of double

Awaiting your valuable suggestions!

ikegami · Accepted Answer · 2012-12-19 01:02:49Z

11

32 arrays * 8 Mi-doubles/array * 8 bytes/double = 2 GiB.

32-bit processes on Windows only have 2GiB of usable address space. Your raw data would take up all available address space, leaving nothing for the data structure's overhead, perl, your program and other variables. The only way it's going to fit in memory is if you switch to a 64-bit Perl. Otherwise, you will have to contend with a necessarily slower solution.

edited Dec 19, 2012 at 1:02

answered Dec 19, 2012 at 0:55

ikegami

391k17 gold badges291 silver badges555 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

InnovWelt Over a year ago

Yes. I could understand the problem now. So I am looking for any other intermediate solution (like Inline::C, Inline. Is it possible to store the data globally in Inline::C ?

ikegami Over a year ago

No, the raw data takes more memory that you have. Switching language is not going to help. You need to switch to a 64-bit process or figure out a way of not having all the data in memory at once.

Leon Timmermans · Accepted Answer · 2012-12-19 00:42:27Z

3

This will create a perl value for each element, which will be quite heavy. You might want to look at something that stores the values as doubles, such as Tie::CArray or Tie::Array::PackedC.

answered Dec 19, 2012 at 0:42

Leon Timmermans

30.3k2 gold badges65 silver badges111 bronze badges

1 Comment

Leon Timmermans Over a year ago

Also, you might not want to create millions of temporary values in the process. Initializing in a loop might be much more feasible.

Sinan Ünür · Accepted Answer · 2012-12-19 15:48:55Z

2

Here is what I get on my Windows XP SP3 system using ActiveState Perl 5.14.2. Task manager was showing Commit charge: 778M/3958M when I ran the following script:

#!/usr/bin/env perl

use strict;
use warnings;

use Devel::Size qw(total_size);

my $unit = 1024 * 1024;
my $topj = (8 * $unit) - 1;
my @data;

for my $i (0 .. 31) {
    print "$i: ";
    my @row;
    $#row = $topj;
    for my $j (0 .. $topj) {
        $row[$j] = 0.25;
    }
    push @data, \@row;
    printf "%.0f\n", total_size(\@data)/$unit;
}

Output:

C:\temp> yy
0: 224
1: 448
2: 672
3: 896
4: 1120
5: 1344
6: 1568
7: 1792
Out of memory!
8:

On the other hand, the following C program does better:

#include <stdlib.h>
#include <stdio.h>

#define ROWSIZE 8*1024*1024

int main(void) {
    int n = 1;
    while (calloc(ROWSIZE, sizeof(double))) {
        printf("%d: success!\n", n);
        n += 1;
    }
    return 0;
}

Output:

1: success!
2: success!
3: success!
…
26: success!
27: success!

for the same 1.7GB footprint.

answered Dec 19, 2012 at 15:48

Sinan Ünür

118k15 gold badges201 silver badges347 bronze badges

1 Comment

InnovWelt Over a year ago

I am looking for any other intermediate solution (like Inline::C, Inline ). Is it possible to store the data globally in Inline::C ?

Collectives™ on Stack Overflow

Need advice for Memory usage in perl

3 Answers 3

2 Comments

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related