using async correctly (with forEach)

Question

So I have a nice chunk of nested async code running together and it all seems to be okay, except for when I get to the end of it. my last function in a series block is a forEach: that then goes into a async.parallel

Managed to track down the following chunk which does not run in order:

async.forEach(array, function(elem, callback) {
      async.parallel([
          function(callback) {
              database-call-A(elem, unction(err, data)  {
                    if(err){
                        console.log("error on first parallel");
                        callback({err: false}); // will break out
                    } else {
                        elem.c = data;
                        callback();
                    }
              });
          },
          function(callback) {
               database-call-B(elem, function(err, data)  {
                    if(err){
                        console.log("error on first parallel");
                        callback({err: false}); // will break out
                    } else {
                        elem.c = data;
                        callback();
                    }
              });
          } 
       ]); // end async.parallel 

       // if forEach needs a callback after every iteration (which I think it does?)
       console.log("PRINTS N TIMES - ONCE FOR EVERY ITERATION");
       callback(); // both parallel functions have run, call back forEach

 }); // end forEach

 console.log("Donions - prints when finished");

When I tested this code by throwing print statements everywhere, I noticed that "PRINTS N TIMES ..." ran N times, then I got "Donions .." and THEN my do something(); and other stuff(); started getting called in my async.parallel.

Why is my forEach iterating through without waiting for the callbacks from async.parallel?

jeremija · Accepted Answer · 2013-12-17 12:44:19Z

4

First, there is no async.forEach, but there is the async.each function. The async.parallel function will execute all tasks simoultaneously and then execute the callback if it was defined.

In your source, the callback function will be executed as soon as the async.parallel is executed, not after all of it's functions have returned. I suggest you read the documentation.

If you want to execute the callback after all the parallel functions' callbacks have been called, you should pass the forEach's callback to the async.parallel function as a second parameter, after the array of functions. Also, you should probably pass the callbacks to both databaseCallA and databaseCallB thus making them call their callbacks when they finish so they aren't executed prematurely.

async.each(array, function(elem, callback) {
    async.parallel([
        function(cb) {
            databaseCallA(cb);
            // if you have called the cb here, the `async.parallel` function would "think" that the databaseCallA has finished, even though it may not have started yet.
        },
        function(cb) {
            databaseCallB(cb):
        } 
    ], callback);
});

You should modify your database call functions to accept the callback argument and make them call the callback after they have done their work.

The point of async calls is to free the resources for other jobs while waiting for your job to finish - your code will continue it's execution, but you can use the callbacks or some kind of event-based notifications to notify others your async job has finished.

EDIT

To execute something after all parallel calls have finished:

async.each(array, function(elem, callback) {
    async.parallel([
        function(cb) {
            // I use the setTimeout function to intentionally
            // delay the response
            setTimeout(function() {
                // the first cb parameter is error, the second is result 
                // in case there was no error
                cb(undefined, 'a');
            }, 500);
        },
        function(cb) {
            setTimeout(function() {
                cb(undefined, 'b');
            }, 300);
        },
        function(cb) {
            setTimeout(function() {
                cb(undefined, 'c');
            }, 800);
        }
    ], function(err, results) {
        // this will be executed only after all three cb's are executed. In
        // this case, after about 800ms. The results variable will be an 
        // array containing the resuts passed to each cb. In this case it
        // will  be ['a', 'b', 'c'].

        // you could call the main callback here
        callback();
    });
}, function(err) {
    // this callback will be executed either after everything was done,
    // or if an error has occurred.
    if (err) {
        handleError(err);
        return;
    }

    // this should be executed after all `each` and `parallel` calls
    // have finished. In this case, it should also be after about 800ms
    // from the start, since everything was executed in parallel
    console.log('finished');

    // now you are sure everything was done, do something afterwards
    after();
});

// this will be executed immediately so don't put anything here
// that depends on the outcome of the async calls above
console.log('test');

edited Dec 17, 2013 at 12:44

answered Dec 17, 2013 at 11:32

jeremija

2,5581 gold badge21 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

gadu Over a year ago

you are right! I simplified a bit too much here, check update for little more info (my db calls are only calling back once they return so we're good on that front) - I'm pretty sure forEach is actually a function considering its run on my computer but I have wondered why I can never find it on the API - let's see if each does anything..

jeremija Over a year ago

so what's exactly your problem?

gadu Over a year ago

exactly what I described, i'm having code AFTER my parallel run before my parallel - so my code gets stuck since it won't return out of forEach because those callbacks have already happened.

jeremija Over a year ago

This is from the documentation:

Note, that since this (each) function  applies the iterator to each item in parallel, there is no guarantee that the iterator functions will complete in order.

You could try with eachSeries.

gadu Over a year ago

I don't need them to run in any order :) Just need parallel to complete before the calls outside of it (but at the very end of .each). I'll try to play around with different iterators, thanks anyways.

|

Sayris · Accepted Answer · 2013-12-17 11:54:22Z

0

I'm not sure for this answer, but according with async documentation, I think you must do this :

async.forEach(array, function(elem, callback) {
    async.parallel([
      function(callback) {
          database-call-A();
          callback();
      },
      function(callback) {
          database-call-B():
          callback(); // callback for second parallel function
      } 
   ], function(err,result) {
         // this code is executed when paralles functions ends
         console.log("PRINTS N TIMES - ONCE FOR EVERY ITERATION");
         callback(); 
   }); // end async.parallel 

 }); // end forEach

regards.

answered Dec 17, 2013 at 11:54

Sayris

1,07810 silver badges10 bronze badges

1 Comment

jeremija Over a year ago

Database calls are (should be) async, and in that situation your callbacks would be executed before the database calls have actually done some work.

Collectives™ on Stack Overflow

using async correctly (with forEach)

2 Answers 2

6 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related