How to do a massive random update with MongoDB / NodeJS

Question

I have a mongoDB collection with more then 1000000 documents and i would like to update each document one by one with a dedicated information (each doc has an information coming from an other collection).

Currently i'm using a cursor that fetch all the data from the collection and i do an update of each records through the async module of Node.js

Fetch all docs :

inst.db.collection(association.collection, function(err, collection) {
    collection.find({}, {}, function(err, cursor) {
        cursor.toArray(function(err, items){
                 ......
        );
    });
});

update each doc :

items.forEach(function(item) {
    // *** do some stuff with item, add field etc.
    tasks.push(function(nextTask) {
       inst.db.collection(association.collection, function(err, collection) {
           if (err) callback(err, null);
           collection.save(item, nextTask);
       });
    });
});

call the "save" task in parallel

async.parallel(tasks, function(err, results) {
    callback(err, results);
});

Ho would you do this type of operation in a more efficient way? I mean how to avoid the initial "find" to load a cursor. Is there now way to do an operation doc by doc knowing that all docs should be updated?

Thanks for your support.

do you have a reason for fetching all of the documents before starting processing? Why not just iterate over a cursor rather than reading all of them into an array and iterating over an array? — Asya Kamsky
– Asya Kamsky, Commented Jan 19, 2013 at 15:11
Yes I agree with your point but still, the cursor will have to be loaded first before processing each doc. Do you have a different example ? — Michael
– Michael, Commented Jan 20, 2013 at 8:41
not really true - the documents will be fetched from the server in batches (by default 100 at a time). You are forcing fetching of all the documents at once. — Asya Kamsky
– Asya Kamsky, Commented Jan 20, 2013 at 15:58

mjhm · Accepted Answer · 2013-01-22 04:41:11Z

You're question inspired me to create a Gist to do some performance testing of different approaches to your problem.

Here are the results running on a small EC2 instance with the MongoDB at localhost. The test scenario is to uniquely operate on every document of a 100000 element collection.

108.661 seconds -- Uses find().toArray to pull in all the items at once then replaces the documents with individual "save" calls.
99.645 seconds -- Uses find().toArray to pull in all the items at once then updates the documents with individual "update" calls.
74.553 seconds -- Iterates on the cursor (find().each) with batchSize = 10, then uses individual update calls.
58.673 seconds -- Iterates on the cursor (find().each) with batchSize = 10000, then uses individual update calls.
4.727 seconds -- Iterates on the cursor with batchSize = 10000, and does inserts into a new collection 10000 items at a time.

Though not included, I also did a test with MapReduce used as a server side filter which ran at about 19 seconds. I would have liked to have similarly used "aggregate" as a server side filter, but it doesn't yet have an option to output to a collection.

The bottom line answer is that if you can get away with it, the fastest option is to pull items from an initial collection via a cursor, update them locally and insert them into a new collection in big chunks. Then you can swap in the new collection for the old.

If you need to keep the database active, then the best option is to use a cursor with a big batchSize, and update the documents in place. The "save" call is slower than "update" because it needs to replace whole document, and probably needs to reindex it as well.

Awsome, thanks for this answer. It clearly explain all options, thanks!

Collectives™ on Stack Overflow

How to do a massive random update with MongoDB / NodeJS

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related