Node.js read large files directory with find and head

Question

In node.js I read files with a spawn of find instead of using fs.readdir that is much more slower:

scanDirStream: function (needle, params) {
            var options = {
                type: 'f',
                name: '*'
            };
            for (var attrname in params) { options[attrname] = params[attrname]; }
            return new Promise((resolve, reject) => {
                var opt = [needle];
                for (var k in options) {
                    var v = options[k];
                    if (!Util.empty(v)) {
                        opt.push('-' + k);
                        opt.push(v);
                    }
                };
                var res = '';
                var find = spawn('find', opt)
                find.stdout.on('data', _data => {
                    var buff = Buffer.from(_data, 'utf-8').toString();
                    if (buff != '') res += buff;
                })
                find.stderr.on('data', error => {
                    return reject(Buffer.from(error, 'utf-8').toString());
                });
                find.on('close', (_) => {
                    data = res.split('\n');
                    data = data.slice(0, data.length - 1);
                    return resolve(data);
                });
            });
        }//scanDirStream

and you call it like

scanDirStream(path, { name: name }).then(files => console.log(files))

For huge directories I would furtherly improve it limiting the output of find, before returning the listing to Node.js, so using a pipe to head like:

find -type -f "*.mp3" | head -n 100

that it should be like

var head = spawn('head',['-n','100])
var find = spawn('find', opt)
head.stdin.pipe(find)
                find.stdout.on('data', _data => {
                    var buff = Buffer.from(_data, 'utf-8').toString();
                    if (buff != '') res += buff;
                })
                find.stderr.on('data', error => {
                    return reject(Buffer.from(error, 'utf-8').toString());
                });
                find.on('close', (_) => {
                    data = res.split('\n');
                    data = data.slice(0, data.length - 1);
                    return resolve(data);
                });

but the pipe does not work properly

when you say fs.readdir is slow, how many files were in the directory? — Rick Su
– Rick Su, Commented Apr 29, 2019 at 13:06
@Neverever we talk about thousand hundreds files at least. There is a known issue in node.js - see here stackoverflow.com/questions/53125592/… — loretoparisi
– loretoparisi, Commented Apr 29, 2019 at 13:41

Rick Su · Accepted Answer · 2019-04-30 14:52:37Z

1

The piping should go from find.stdout to head.stdin, and final output is coming out of head.stdout

So, the code should look like this:

find.stdout.pipe(head.stdin);

And attach the on data & close listener on head

head.stdout.on('data', ...);

head.on('close', ...);

For error handling, you could attach listener to both find.stderr and head.stderr, since error could occur from any/both of the two commands.

answered Apr 30, 2019 at 14:52

Rick Su

16.5k3 gold badges35 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

loretoparisi Over a year ago

So thank you. It apparently works, but it seems it is not doing the head -n N when doing var head = spawn('head',[ '-n', options.limit]); - see here gist.github.com/loretoparisi/6e479d6b1bb4990af649cf3308b8b3c2

Rick Su Over a year ago

what's the opt printed value? and is head printing all or none?

loretoparisi Over a year ago

So the parameters of find are opt=[ '/root/', '-type', 'f', '-name', '*.wav' ] while the parameters of head are like ['-n','10'] to set the limit let's say to 10. I get the listing of files, but it seems the | head has not been applied, since I get all files, not the head.

Collectives™ on Stack Overflow

Node.js read large files directory with find and head

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related