mongodb 2.4.1 failing with 'process out of memory'

Question

I have the following mongo versions

db version v2.4.1    
MongoDB shell version: 2.4.1,

and

db version v2.2.1-rc1, pdfile version 4.5, 
MongoDB shell version: 2.2.1-rc1

installed on 64-bit windows 7 machine.

I have a collection having 10001000 (10 million+) records, when I use V 2.4.1 to aggregate, it fails with the following error:

Fatal error in CALL_AND_RETRY_2
Allocation failed - process out of memory

However when I use V 2.2.1-rc1, to aggregate the same collection, it works fine and gives result in around 1 minute.

Sample document of the collection that is being aggregated:

{

    "_id" : ObjectId("516bdd1c39b10c722792e007"),
    "f1" : 10000010,
    "f2" : 10000000,
    "key" : 0
}

Aggregation Command:

{$group: {"_id": "$key", total: {$sum: "$f1"}}}

Command used to populate records:

for(var i = 10011000; i < 10041000; ++i)
{ 
    db.testp.insert({"f1": i+10, "f2": i, "key": i%1000})
}

Ross · Accepted Answer · 2013-04-25 10:18:08Z

4

How much memory do you have? Could it be the $group is taking up more than 10% of available memory and causing the error? See the aggregation documentation on memory for cumulative operators.

edit 1:

Out of interest - does the aggregation work outside the shell? eg calling it from a driver. I have seen similar v8 errors and as the shell was updated to v8 in 2.4 Theres a chance it could be that.

edit 2:

If the resulting array is too big in the shell then that can also trigger the error: see SERVER-8859. To work around you might need to run multiple aggregations, either by doing a $match early on to limit the working set or even a $skip and $limit to paginate through the result set.

I tried your aggregation with 10,070,999 docs on 2.4.1 on a mac and didn't get the error

edited Apr 25, 2013 at 10:18

answered Apr 25, 2013 at 9:44

Ross

18.1k2 gold badges59 silver badges76 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

ranjjose Over a year ago

Ross, Thanks for the reply. I have 4GB RAM in my system. I had tested with 100,1000, 100000 unique keys (i.e., "_id" = "$key" in $group) It failed in all those cases on V2.4. However it worked fine in V2.2. I did read about 10% memory utilization thing earlier; but the same is present in V2.2 as well. But it works fine on V2.2!! The only difference I could read was something to do with $sort in Memory for Cumulative Operators. I'm not sorting here anyways.

Ross Over a year ago

I'd try it using a driver and see if you are hitting a V8 shell issue.

ranjjose Over a year ago

Thank you for your time. About edit1, I tried the same through pymongo and it worked. I got a reply here as follows:

Try using the v2.2 shell to connect to the v2.4 mongo db, and see if the error reproduces. I suspect that this error is coming from the shell, not from MongoDB.

I tried that. That is mongod V2.4 and mongo V2.2. That too worked! Thanks again!

ranjjose Over a year ago

From whatever I see, wherever mongo 2.4 shell is involved it fails!

Ross Over a year ago

Ok well if it works through the driver and it works with the old shell (2.2) then it is a manifestation of SERVER-8859. If you could provide a full test case that would be really helpful in ensuring it does get fixed and whether it is the same issue or just has the same result. As I said using your test code I couldn't reproduce.

|

Collectives™ on Stack Overflow

mongodb 2.4.1 failing with 'process out of memory'

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related