Best practice for making async API calls in node using fetch? [closed]

Question

Closed. This question is opinion-based. It is not currently accepting answers.

Want to improve this question? Because this question may lead to opinionated discussion, debate, and answers, it has been closed. You may edit the question if you feel you can improve it so that it requires answers that include facts and citations or a detailed explanation of the proposed solution. If edited, the question will be reviewed and might be reopened.

Closed 2 years ago.

Improve this question

I'm building an app in nodeJS, express and MySQL where I'm collecting data from several different API:s using node fetch, one which has a limit of 200 calls per 5 minute.

In order to update my database completely I need to essentially run calls to this API nonstop, currently I'm trying to stay within the boundaries of the limit by using a sleep in a for loop/foreach.

The problem is that each request can vary in time and it just seems like I could make the calls more effective rather than constantly sleeping the same amount of time for each request. I was wondering if there's a better way to do this? I've been thinking about making e.g 199 calls async and then waiting exactly 5 minutes, and then repeating it.. but I have no idea how.

Couldn't find a thread on this but please do link one if there's a good answer already. Answers are much appreciated, and if you have other feedback please do let me know!

  for (let i = 1; i < length; i++) {
    try {
      let id = IdCollection[i];
      let resultFromApiCall = await datafetcher.MakeApiCall(id);

      await sleep(2000);

    } catch (error) {
      console.log("Error in api-calls" + error);
    }
  }```

I think the optimal solution depends on what you're fetching here. I don't know if there needs to be that delay between each fetch or not, because I don't know what you're fetching. — Robo Robok
– Robo Robok, Commented May 4, 2020 at 22:52
If you absolutely need to make that many requests perhaps you should look into WebSockets (html5rocks.com/en/tutorials/eventsource/basics) — Joeri
– Joeri, Commented May 4, 2020 at 22:55

htho · Accepted Answer · 2020-05-07 05:30:29Z

1

I think it is a matter of taste here.

I'd make all allowed requests at once and then wait until the 5min interval is over.

  // constants to define the restrictions
  const restrictionCalls = 5;
  const restrictionTime = 300000; // 5 * 60 * 1000

  let throttleTimer = Date.now(); // when did we start this interval?
  let throttleCounter = 0; // how many calls did we make in this interval?

  for (let i = 1; i < length; i++) {
    try {
      let id = IdCollection[i];
      throttleCounter++;
      let resultFromApiCall = await datafetcher.MakeApiCall(id);
    } catch (error) {
      console.log("Error in api-calls" + error);
    }

    if(throttleCounter === restrictionCalls) {
      const timePassed = Date.now() - throttleTimer;
      const waitTime = restrictionTime - timePassed;
      if(waitTime > 0) await sleep(waitTime);

      // reset
      throttleTimer = Date.now();
      throttleCounter = 0;
    }
  }

edited May 7, 2020 at 5:30

answered May 5, 2020 at 4:36

htho

1,8921 gold badge17 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Thorvald Over a year ago

Thank you! Trying this out and it seems to work great with some minor tweaks. Brilliant :)

Collectives™ on Stack Overflow

Best practice for making async API calls in node using fetch? [closed]

1 Answer 1

1 Comment

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Related