NodeJS read huge directory file by file

Question

I have a directory composed of empty files and I want to iterate through each of their names. However I have nearly 20 million of these empty files and to load them all into memory with fs.readdir or fs.readdirSync would both take needlessly long and devour all my memory on the system.

What would be a way to go about this?

Ideally I would look for something that reads file by file in an async fashion with code that would ressemble the following:

readdirfilebyfile((filename)=>{....}) so that at no point would I keep the entire list of files in memory.

The current solution I am using is dumping all the file names into a single file which I then read as a data stream. However, this is just running away from a problem that I should know how to solve without resorting to this.

Duplicate? stackoverflow.com/questions/25757293/… There also is groups.google.com/forum/#!topic/nodejs/t0ziBVsPRqw There are some indirect solutions at least in the second link. There is no direct solution purely within node.js I think. But the answer in the linked SO thread says it's no problem (and he tried), you just have to ensure not to start processing them all at once, just reading that big list seems to be fine. Okay - 600 MB of memory if I use his calculations as a basis for your scenario.... — Mörre
– Mörre, Commented May 27, 2017 at 18:26
@Mörre I upvoted the question just now because OP didn't get a functional answer. The answer still wants to read all file names into memory. I want something that works by accessing an n file without reading the ones that came before it or after. — Slava Knyazev
– Slava Knyazev, Commented May 27, 2017 at 18:46
What are you trying to accomplish? What action do you want to perform on those filenames? — robertklep
– robertklep, Commented May 27, 2017 at 19:37
That is why I included the 2nd link. Did you have a look at it? There is no "pure" node.js solution. — Mörre
– Mörre, Commented May 27, 2017 at 20:21
@robertklep Pass the names to an arbitrary function that will do things irrelevant to filesystem I/O — Slava Knyazev
– Slava Knyazev, Commented May 27, 2017 at 20:32

dsdenes · Accepted Answer · 2017-05-27 19:32:27Z

1

What about this one? pv is pipe viewer, a rate limiter for bash pipes.

const spawn = require('child_process').spawn;
const exec = require('child_process').exec;
const tail = spawn('tail -f /tpm/filelist | pv -l -L 10 -q');
tail.stdout.on('data', fileName => {
  // parse filenames here
  console.log(fileName);
});
exec('ls > /tpm/filelist');

answered May 27, 2017 at 19:32

dsdenes

1,0356 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

NodeJS read huge directory file by file

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related