Memory usage of Node.js application increases when writing files

Question

I'm connected to a p2p network and receiving messages every some seconds. The messages can have a size of 10 chars but also 5 MB. I need to log all messages I receive by writing them to the file system. At the moment I'm using fs.writeFileSync() but unfortunately, the memory increases the longer my application is running:

In this case, my application is running for 15 hours and uses already 3.7Gi. The function I use for writing my logs is the following:

let counter = 0

class Message {
    constructor(data) {
        this.msgId = data.readInt32LE()
        const length = data.readInt32LE()
        // this.data = JSON.parse(data.read(<length>))
    }
    
    // some other methods

    save() {
        if (!fs.existsSync(`/data/${this.msgId}`)) {        
            const data = Object.assign(this.data, {
              index: counter++
            })
            console.log(counter)
            fs.writeFileSync(`/data/${this.msgId}`, JSON.stringify(data))
          }
    }
}

Whenever I receive a message over p2p, I'm reading the msgId and the data it contains within the constructor. After doing some other business logic on the "Message" object, I finally calling the save() function. The "memory leak" is definitely inside the save() function because when I leave the save() function empty, my application runs for days without cosnuming more than 200MB memory. So does anyone know if fs.writeFilesync() has a potential memory leak or what might cause the memory leak?

Something is fishy here - are you actually passing data, which is an object, to fs.writeFileSync? — Robert Kawecki
– Robert Kawecki, Commented Mar 29, 2021 at 11:38
Yeah, sorry... there's another function (I removed to shrink down the excerpt) that is called before the save() function. It's doing some business logic and finally does this.data = JSON.stringify(this.data). I fixed the missing JSON.stringify() above.... — nrgyzer
– nrgyzer, Commented Mar 29, 2021 at 14:24

Mark D · Accepted Answer · 2021-04-03 18:34:36Z

1

I am having the same issue while using node 15.8.0 on windows 10.

I replaced fs.writeFileSync(files[i], data, { encoding: 'utf8' }); with

let fd = 0;
try {
   fd = fs.openSync( files[i], 'w', 0o666  );  
   let writen = fs.writeSync( fd, data, 0, 'utf8' );
} catch ( e ) {
} finally {
   if ( fd )
     fs.closeSync(fd);
}

This resolved the issue with expanding heap.

edited Apr 3, 2021 at 18:34

answered Apr 2, 2021 at 21:13

Mark D

262 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

nrgyzer Over a year ago

Yeah, this reduced the memory usage dramatically. So I can write 1 Mio. messages which was impossible previously.

zzzgoo Over a year ago

Hi, could you explain what's the cause of the issue?

Collectives™ on Stack Overflow

Memory usage of Node.js application increases when writing files

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related