0

I am trying to fill mongoDB database with some random data from faker. So i used setInterval function to run a database Query over and over its working but the problem is when i monitor my memory usage it just growing and growing until it reaches to the that limit of v8 engine no matter how much i increase the limits it crashes anyway just a matter of time.

const faker = require("faker")
let userName; 
let email; 

setInterval(async () => {
  userName = faker.name.findName(); 
  email = faker.internet.email(); 
  await User.create({
    userName: userName,
    email: email,
  })
    .then(() => {
      //getting memory usage
      console.log(process.memoryUsage().heapUsed / 1024 / 1024);
    })
    .catch(err => {
      console.log(err);
    });
});

how should i manage memory allocation to stop crashes? is this considered to be a memory leak?

1 Answer 1

1

Calling setInterval with no time, this is adding to the callback queue as fast as possible, without any limits. And if you add items to a queue faster than you can remove them, memory will always grow.

*Edit: to understand exactly what happens with the event loop and callback queue when you do a setInterval, I recommend watching the excellent JSConf talks by Philip Roberts and/or Jake Archibald.

You have a few options. You could change to a setTimeout, and then invoke another setTimeout in your .then, once the last call has finished. This would essentially serialize it.

A more nuanced approach would be to limit the max number of concurrent mongoDB writes by using a counter. That way you can have a certain number of writes happening concurrently. In this example, it limits the number of concurrent writes to 10. This should keep your memory from growing without limits.

const faker = require("faker")
let userName; 
let email; 
let numRunning = 0;
let maxRunning = 10;

setInterval(async () => {
  if (numRunning > maxRunning) return;
  numRunning++;
  userName = faker.name.findName(); 
  email = faker.internet.email(); 
  await User.create({
    userName: createdUser.userName,
    age: createdUser.age,
  })
    .then(() => {
      numRunning--;
      //getting memory usage
      console.log(process.memoryUsage().heapUsed / 1024 / 1024);
    })
    .catch(err => {
      numRunning--;
      console.log(err);
    });
}, 50);

I also added an interval of 50ms. A zero for a setInterval really isn't a good idea. There are good reasons to sometimes use a zero for setTimeout, but I can't think of a good reason to ever use a zero for a setInterval.

Sign up to request clarification or add additional context in comments.

1 Comment

thanks @David784 yeah isee there was a misunderstanding in how setInterval actually works

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.