Writing large file in Nodejs using fs.createWriteStream

Question

I have function, which in perfect world should create an huge 1M lines file. Here it is:

    const fileWriteStream = fs.createWriteStream(path.resolve(filePath));
    let ableToWrite = true;
    for (let i = 0; i < 1e6; i++) {
        if (ableToWrite) {
            ableToWrite = fileWriteStream.write(`.testClass${itr}-${i%2} { background: red } \n`);
        } else {
            fileWriteStream.once('drain', () => {
                ableToWrite = fileWriteStream.write(`.testClass${itr}-${i%2} { background: red } \n`);
            })
        }
    }

Unfortunetly for me, i'm getting the following error pretty fast:

MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added. Use emitter.setMaxListeners() to increase limit

I don't really want to increase listeners count for this function. What is correct way to write such a big file using streams?

Many thanks!

You're going to need another means of flow control: a recursive function, generator, etc. — Fissure King
– Fissure King, Commented May 21, 2018 at 20:36
Do you understand why your current code adds 99999 listeners? — Bergi
– Bergi, Commented May 21, 2018 at 21:19
@Bergi To be honest not really. I know that i have limit for subscribtions, but it looks like this code only add new watchers and watchers which has already been added not work at all — Yafim Dziuko
– Yafim Dziuko, Commented May 22, 2018 at 7:29
@YafimDziuko The problem is that it doesn't wait after adding the listeners. So in the following iterations (ableToWrite still being false) it continues to add all the listeners - everything before even the first event fires. And when it fires, it fires all the listeners waiting for it. — Bergi
– Bergi, Commented May 22, 2018 at 9:13

Bergi · Accepted Answer · 2018-05-21 21:27:26Z

5

The easiest way to asynchronously continue when the stream drains is to use await in the loop with a promise:

const fileWriteStream = fs.createWriteStream(path.resolve(filePath));
for (let i = 0; i < 1e6; i++) {
    const ableToWrite = fileWriteStream.write(`.testClass${itr}-${i%2} { background: red } \n`);
    if (!ableToWrite) {
        await new Promise(resolve => {
            fileWriteStream.once('drain', resolve);
        });
    }
}

The alternative is using a recursive approach instead of a loop:

function go(i) {
    if (i >= 1e6) return;
    const ableToWrite = fileWriteStream.write(`.testClass${itr}-${i%2} { background: red } \n`);
    if (ableToWrite)
        go(i+1);
    else
        fileWriteStream.once('drain', () => {
            go(i+1);
        });
}
const fileWriteStream = fs.createWriteStream(path.resolve(filePath));
go(0);

answered May 21, 2018 at 21:27

Bergi

671k162 gold badges1k silver badges1.5k bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Patrick Roberts Over a year ago

would await new Promise(fileWriteStream.once.bind(fileWriteStream, 'drain')) work? Not sure if that's a good idea since that might implicitly do something with reject, but I've never tested that.

Bergi Over a year ago

@PatrickRoberts I guess it would work as once ignores a third argument, but I still don't think it's a good idea because the arrow function is simply much more readable.

Yafim Dziuko Over a year ago

@Bergi First solution works really fine, thx! But can you please explain one thing - until new Promise wont be resolved, for loop will be paused? Looks like it shoud, because line order in file seems correct

Yafim Dziuko Over a year ago

@Bergi Tested it, await new Promise really pausing for loop untill it won't be resolved. Wow! Big thanks for your help

Collectives™ on Stack Overflow

Writing large file in Nodejs using fs.createWriteStream

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related