Weird memory usage on Node.js

Question

This simple code stores 1 million strings (100 chars length) in an array.

function makestring(len) {
    var s = '';
    while (len--) s = s+'1';
    return s;
}

var s = '';
var arr = [];
for (var i=0; i<1000000; i++) {
    s = makestring(100);
    arr.push(s);
    if (i%1000 == 0) console.log(i+' - '+s);
}

When I run it, I get this error:

(...)
408000 - 1111111111111111111 (...)
409000 - 1111111111111111111 (...)
FATAL ERROR: JS Allocation failed - process out of memory

That's strange 1 million * 100 are just 100 megabytes.

But if I move the s = makestring(100); outside the loop...

var s = makestring(100);
var arr = [];
for (var i=0; i<1000000; i++) {
    arr.push(s);
    if (i%1000 == 0) {
        console.log(i+' - '+s);
    }
}

This executes without errors!

Why? How can I store 1 Million objects in node?

Sirko · Accepted Answer · 2013-01-28 07:45:40Z

2

In the moment you move the String generation outside the loop, you basically just create one String and push it a million times into the array.

Inside the array, however, just pointers to the original String are used, which is far less memory consuming then saving the String a million times.

answered Jan 28, 2013 at 7:45

Sirko

74.3k19 gold badges157 silver badges194 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Nevir · Accepted Answer · 2013-01-28 07:46:41Z

0

Your first example builds 1000000 strings.

In your second example, you're taking the same string object and adding it to your array 1000000 times. (it's not copying the string; each entry of the array points to the same object)

V8 does a lot of things to optimize string use. For example, string concatenation is less expensive (in most cases) than you think. Rather than building a whole new string, it will typically opt to connect them i a linked list fashion under the covers.

answered Jan 28, 2013 at 7:46

Nevir

8,1014 gold badges43 silver badges50 bronze badges

5 Comments

Peter Over a year ago

anyway, 1 Millon strings (~100MB) shouldn't eat all memory

Nevir Over a year ago

Your math is off. For one, each String has a fair amount of overhead by V8 for bookkeeping (I'm not sure how much - probably on the order of 10 bytes per string). I'm also fairly certain that V8 stores strings in UTF-16 by default (2 bytes per char).

Nevir Over a year ago

If you want a more memory efficient way of storing large quantities of data, take a look at Node's Buffer class.

Peter Over a year ago

Nevir, even if the overhead is 500 bytes per string, 1 Million would be 600 MB, far below the memory limit.

Nevir Over a year ago

Note that in your example, your makestring function is creating 100 new strings each time (each with a certain amount of overhead), and V8 is probably not flattening them into one object. Chances are that your example actually built 1000000 * 100 string objects.

Collectives™ on Stack Overflow

Weird memory usage on Node.js

2 Answers 2

Comments

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related