Mongodb Aggregation Map reduce

Question

I have started exploring mongodb couple of weeks back. I have a scenario here. I have a collection which has 3 million records.

I would want to perform aggregation on the aggreation based on two keys (also need to use match condition). I used aggregation framework for the same. I came to know that aggregation would fail if the processing document size (array) exceeds 16 MB.

I faced the same issue when i tried. I am trying to use map reduce now. I would need the guidance on implementing the same. How can I overcome the 16 MB size limit by using map reduce?

Also I came to know that I can do it by splitting the collection into multiple collections and do the aggregation on the same. Would be great if anyone can point me in right direction?

Could you please provide some sample data and what query you are executing. — Lalit Agarwal
– Lalit Agarwal, Commented Apr 16, 2014 at 10:59

Neil Lunn · Accepted Answer · 2014-04-16 11:26:45Z

Even without code there are basic answers to your questions.

The limitation on the BSON document 16MB output size is for "inline" responses. That means a response from your operations that does not write the individual "documents" from your response to a collection.

So with mapReduce a statement much like this:

db.collection.mapReduce(
    mapper,
    reducer,
    { "out": { "inline": 1 } }
)

Has the problem that the "array" in the response needs to be under 16MB. But if you change this to output to a collection:

db.collection.mapReduce(
    mapper,
    reducer,
    { "out": { "replace": "newcollection" } }
)

Then you no longer have this limitation.

The same applies to the aggregate method from versions 2.6 and upwards using the $out pipeline stage:

db.collection.aggregate([
   // lots of pipeline

   { "$out": "newcollection }

])

This overcomes the limtation by the same means by outputing to a collection.

Actually with the aggregate statement, again from version 2.6 and upwards this returns a cursor, just like the .find() method, and is also not subject to this limitation.

Collectives™ on Stack Overflow

Mongodb Aggregation Map reduce

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related