node.js and express : Sequential execution flow one mongodb query request after another

Question

I have a webserver running in node.js and Express which retrieves data from mongodb . In mongodb collections are getting created dynamically and the name of newly created collection will be stored in one metadata collection “project” . My requirement is to firstly iterate to metadata collection to get the collection name and then get inside the each collection to do multiple query based on some condition . Because my collection metadata is dynamic I have tried to do using for loop . But it is giving wrong data . It is not executing sequent . Before finishing the loop execution it is returning the value .How to perform sequential execution in node.js using node core modules only (Not other library like async..);

exports.projectCount = function (req, res) {
    var mongo = require("mongodb"),
        Server = mongo.Server,
        Db = mongo.Db;

    var server = new Server("localhost", 27017, {
        auto_reconnect: true
    });

    var db = new Db("test", server);
   // global JSON object to store manipulated data 
    var projectDetail = {
        projectCount: 0,
        projectPercent: 0
    };
    var totalProject = 0;
    db.open(function (err, collection) {
        //metadata collection 
        collection = db.collection("project");
        collection.find().toArray(function (err, result) {
            // Length of metadata  collection 
            projectDetail.projectCount = result.length;
            var count = 0;
            //iterate through each of the array which is the name of collection 
            result.forEach(function (item) {
                //change collection  object  to new collection
                collection = db.collection(item.keyParameter.wbsName);
                // Perform first query based on some condition 
                collection.find({
                    $where: "this.status == 'Created'"
                }).toArray(function (err, result) {
                    // based on result of query one increment the value of count 
                    count += result.lenght;
                    // Perform second query based on some condition 
                    collection.find({
                        $where: "this.status=='Completed'"
                    }).toArray(function (err, result) {
                        count += result.length;
                    });
                });
            });
            // it is returning the value without finishing the above manipulation
            // not waiting for above callback and value of count is coming zero .
            res.render('index', {
                projectDetail: projectDetail.projectCount, 
                count: count
            });

        });
    });
};

fardjad · Accepted Answer · 2013-10-04 08:55:02Z

9

When you want to call multiple asynchronous functions in order, you should call the first one, call the next one in it's callback and so on. The code would look like:

asyncFunction1(args, function () {
  asyncFunction2(args, function () {
    asyncFunction3(args, function () {
      // ...
    })
  })
});

Using this approach, you may end up with an ugly hard-to-maintain piece of code.

There are various ways to achieve the same functionality without nesting callbacks, like using async.js or node-fibers.

Here is how you can do it using node.js EventEmitter:

var events = require('events');
var EventEmitter = events.EventEmitter;

var flowController = new EventEmitter();

flowController.on('start', function (start_args) {
  asyncFunction1(args, function () {
    flowController.emit('2', next_function_args);
  });
});

flowController.on('2', function (args_coming_from_1) {
  asyncFunction2(args, function () {
    flowController.emit('3', next_function_args);
  });
});

flowController.on('3', function (args_coming_from_2) {
  asyncFunction3(args, function () {
    // ...
  });
});

flowController.emit('start', start_args);

For loop simulation example:

var events = require('events');
var EventEmitter = events.EventEmitter;

var flowController = new EventEmitter();

var items = ['1', '2', '3'];

flowController.on('doWork', function (i) {
  if (i >= items.length) {
    flowController.emit('finished');
    return;
  }

  asyncFunction(item[i], function () {
    flowController.emit('doWork', i + 1);
  });

});

flowController.on('finished', function () {
  console.log('finished');
});

flowController.emit('doWork', 0);

edited Oct 4, 2013 at 8:55

answered Oct 4, 2013 at 7:33

fardjad

20.5k6 gold badges55 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Sumeet Kumar Yadav Over a year ago

But if code is executing inside for loop I will come to know that all event execution is finished for all and then return .

fardjad Over a year ago

@Sumeet I added a for loop simulation example, this should give you an idea for implementing other iteration structures.

Sumeet Kumar Yadav Over a year ago

flowController.emit('doWork', i + 1) is not Calling the flowController.on('doWork' ) with incrementd value . Also it is not finishing the execution . Output Is only 2. Can you explain me bit more

Peter Lyons · Accepted Answer · 2013-10-04 07:12:11Z

0

Use callbacks or promises or a flow control library. You cannot program servers in node without understanding at the very least one of these approaches, and honestly all halfway decent node programmers thoroughly understand all three of them (including a handful of different flow control libraries).

This is not a something you are going to just get an answer coded for you by someone else on stackoverflow and then move on. This is a fundamental thing you have to go and study and learn generically as it is only going to come up over and over again on a daily basis.

http://howtonode.org/control-flow

http://callbackhell.com/

answered Oct 4, 2013 at 7:12

Peter Lyons

147k32 gold badges285 silver badges281 bronze badges

1 Comment

Sriharsha Over a year ago

Can using Promises(strongloop.com/strongblog/…) here solve the purpose instead of relying on callbacks?

Jorge Olivero · Accepted Answer · 2014-02-11 20:33:27Z

0

Per the resources in the answer above me, nesting the callback when you iterate and only calling it if you are on the last iteration will solve you problem.

answered Feb 11, 2014 at 20:33

Jorge Olivero

3,8639 gold badges29 silver badges34 bronze badges

Collectives™ on Stack Overflow

node.js and express : Sequential execution flow one mongodb query request after another

3 Answers 3

3 Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related