Nodejs Running Functions in Series

Question

So right now I'm trying to use Nodejs to access files in order to write them to a server and process them.

I've split it into the following steps:

Traverse directories to generate an array of all of the file paths
Put the raw text data from each of file paths in another array
Process the raw data

The first two steps are working fine, using these functions:

var walk = function(dir, done) {
    var results = [];
    fs.readdir(dir, function(err, list) {
        if (err) return done(err);
        var pending = list.length;
        if (!pending) return done(null, results);
        list.forEach(function(file) {
            file = path.resolve(dir, file);
            fs.stat(file, function(err, stat) {
                if (stat && stat.isDirectory()) {
                    walk(file, function(err, res) {
                        results = results.concat(res);
                        if (!--pending) done(null, results);
                    });
                } else {
                    results.push(file);
                    if (!--pending) done(null, results);
                }
            });
        });
    });
};
function processfilepaths(callback) {
    // reading each file
    for (var k in filepaths) { if (arrayHasOwnIndex(filepaths, k)) {
        fs.readFile(filepaths[k], function (err, data) {
            if (err) throw err;
            rawdata[k] = data.toString().split(/ *[\t\r\n\v\f]+/g);
            for (var j in rawdata[k]) { if (arrayHasOwnIndex(rawdata[k], j)) {
                rawdata[k][j] = rawdata[k][j].split(/: *|: +/);
            }}
        });
    }}
    if (callback) callback();
}

Obviously, I want to call the function processrawdata() after all of the data has been loaded. However, using callbacks doesn't seem to work.

walk(rootdirectory, function(err, results) {
    if (err) throw err;
    filepaths = results.slice();
    processfilepaths(processrawdata);
});

This never causes an error. Everything seems to run perfectly except that processrawdata() is always finished before processfilepaths(). What am I doing wrong?

Now you must consider using promises to have better control flow. — Nidhin David
– Nidhin David, Commented Mar 1, 2016 at 5:11

gpopoteur · Accepted Answer · 2014-04-07 08:44:06Z

1

You are having a problem with callback invocation and asynchronously calling functions. IMO I'll recommend that you use a library such as after-all to execute a callback once all your functions get executed.

Here's a example, here the function done will be called once all the functions wrapped with next are called.

var afterAll = require('after-all');

// Call `done` once all the functions
// wrapped with next() get called
next = afterAll(done);

// first execute this
setTimeout(next(function() {
  console.log('Step two.');
}), 500);

// then this
setTimeout(next(function() {
  console.log('Step one.');
}), 100);

function done() {
  console.log("Yay we're done!");
}

answered Apr 7, 2014 at 8:44

gpopoteur

1,51910 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

gpopoteur Over a year ago

@CosmoVibe if this answer was right for your needs, could you please mark it as the right answer?

zhuli Over a year ago

Ah, thanks for letting me know. Didn't realize I had to mark it haha

Amol M Kulkarni · Accepted Answer · 2014-04-07 08:22:48Z

1

I think for your problem, you can use async module for Node.js:

async.series([
    function(){ ... },
    function(){ ... }
]);

To answer you actual question, I need to explain how Node.js works:
Say, when you call an async operation (say mysql db query), Node.js sends "execute this query" to MySQL. Since this query will take some time (may be some milliseconds), Node.js performs the query using the MySQL async library - getting back to the event loop and doing something else there while waiting for MySQL to get back to us. Like handling that HTTP request. So, In your case both functions are independent and executes almost in parallel.

For more information:

Async.js for use with Node.js

edited Apr 7, 2014 at 8:22

answered Apr 7, 2014 at 8:17

Amol M Kulkarni

21.7k34 gold badges125 silver badges165 bronze badges

1 Comment

zhuli Over a year ago

Async didn't work for me, it didn't change the order of execution for some reason. I assume it's because the logic is the same as the callback function, so it acts the same way.

user9170 · Accepted Answer · 2015-01-16 13:22:54Z

function processfilepaths(callback) {
    // reading each file
    for (var k in filepaths) { if (arrayHasOwnIndex(filepaths, k)) {
        fs.readFile(filepaths[k], function (err, data) {
            if (err) throw err;
            rawdata[k] = data.toString().split(/ *[\t\r\n\v\f]+/g);
            for (var j in rawdata[k]) { if (arrayHasOwnIndex(rawdata[k], j)) {
                rawdata[k][j] = rawdata[k][j].split(/: *|: +/);
            }}
        });
    }}
    if (callback) callback();
}

Realize that you have:

for
    readfile (err, callback) {... }
if ...

Node will call each readfile asynchronously, which only sets up the event and callback, then when it is done calling each readfile, it will do the if, before the callback probably even has a chance to get invoked.

You need to use either Promises, or a promise module like async to serialize it. What you would then do looks like:

async.XXXX(filepaths, processRawData, 
   function (err, ...) {
      // function for when all are done
      if (callback) callback();
   }
);

Where XXXX is one of the functions from the library like series, parallel, each, etc... The only thing you also need to know is in your process raw data, async gives you a callback to call when done. Unless you really need sequential access (I don't think you do) use parallel so that you can queue up as many i/o events as possible, it should execute faster, maybe only marginally, but it'll better leverage the hardware.

Collectives™ on Stack Overflow

Nodejs Running Functions in Series

3 Answers 3

2 Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related