Bulk insert in MongoDB using mongoose

Question

I currently have a collection in Mongodb say "Collection1". I have the following array of objects that need to be into inserted into MongoDB. I am using Mongoose API. For now, I am iterating through the array and inserting each of them into mongo. This is ok for now, but will be a problem when the data is too big. I need a way of inserting the data in bulk into MongoDB without repetition. I am not sure how to do this. I could not find a bulk option in Mongoose.

My code below

myData = [Obj1,Obj2,Obj3.......]

myData.forEach(function(ele){
      //console.log(ele)
     saveToMongo(ele);
    });
function saveToMongo(obj){
    (new Collection1(obj)).save(function (err, response) {
          if (err) {
             // console.log('Error while inserting: ' + obj.name + " " +err);
          } else {
            // console.log('Data successfully inserted');
          }
      });

      return Collection1(obj);
  }

chridam · Accepted Answer · 2016-05-22 20:51:33Z

38

You might want to use the insertMany() method here if you're using the latest Mongoose version 4.4.X and greater, which essentially uses Model.collection.insertMany() under the hood and the driver might handle parallelizing >= 1000 docs for you.

myData = [Obj1, Obj2, Obj3.......];
Collection1.insertMany(myData, function(error, docs) {});

or using Promises for better error handling

Collection1.insertMany(myData)
    .then(function(docs) {
         // do something with docs
    })
    .catch(function(err) {
        // error handling here
    });

It works by creating a bunch of documents, calls .validate() on them in parallel, and then calls the underlying driver's insertMany() on the result of toObject({ virtuals: false }); of each doc. Although insertMany() doesn't trigger pre-save hooks, it has better performance because it only makes 1 round-trip to the server rather than 1 for each document.

For Mongoose versions ~3.8.8, ~3.8.22, 4.x which support MongoDB Server >=2.6.x, you could use the Bulk API as follows

var bulk = Collection1.collection.initializeOrderedBulkOp(),
    counter = 0;

myData.forEach(function(doc) {
    bulk.insert(doc);

    counter++;
    if (counter % 500 == 0) {
        bulk.execute(function(err, r) {
           // do something with the result
           bulk = Collection1.collection.initializeOrderedBulkOp();
           counter = 0;
        });
    }
});

// Catch any docs in the queue under or over the 500's
if (counter > 0) {
    bulk.execute(function(err,result) {
       // do something with the result here
    });
}

answered May 22, 2016 at 20:51

chridam

104k26 gold badges246 silver badges243 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

John Over a year ago

Hi, I'm trying your way to add lot of data with bulk mongoose (I have 409 584 data to add) but I have just 273001 data added. Do you know why ?

chridam Over a year ago

What's your MongoDB server version?

John Over a year ago

I have MongoDB versio 3.2.9 and mongoose 4.7.2. If I use insertMany I got FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory then I try to use bulk method.

Aseer KT Miqdad Over a year ago

can we preserve the order in which they created

user3227295 · Accepted Answer · 2016-05-22 20:45:08Z

2

you can pass an array of objects to mongoose model create function

var Collection1 = mongoose.model('Collection1');

Collection1.create(myData,function(err){
    if(err) ...
});

answered May 22, 2016 at 20:45

user3227295

2,1561 gold badge13 silver badges17 bronze badges

Collectives™ on Stack Overflow

Bulk insert in MongoDB using mongoose

2 Answers 2

4 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related