How to Run an API Calls in Parallel (Node.js)

Question

I am trying to run some API calls in parallel, but am having problems since I am trying to call a function again before the API data has been returned.

I am thinking that I could possibly use the new command in Node, but am not sure how to structure it into this scheme. I am trying to avoid recursion, as I already have a recursive version working and it is slow.

Currently I am trying to this code on the server.

loopThroughArray(req, res) { 
  for(let i=0; i<req.map.length; i++) {
    stack[i] = (callback) => {
      let data = getApi(req, res, req.map[i], callback)
    }
  }

  async.parallel(stack, (result) => {
      res.json(result)
  })
}

....

function getApi(req, res, num, cb) {
  request({
    url: 'https://example.com/api/' + num
  },
  (error, response, body) => {
    if(error) {
      // Log error
    } else {
      let i = {
        name: JSON.parse(body)['name'],
        age: '100'
      }
      console.log(body) // Returns empty value array.length > 1 (req.map[i])
      cb(i)
    }
  })

Is there a way to spawn new instances of the function each time it's called and accumulate the results to send back as one result to the client?

Terry Lennox · Accepted Answer · 2018-08-09 16:53:36Z

3

Here's an example of calling Web APIs (each with different parameters), using the Async library, we start by creating an array of N function variables.

const async = require('async');
const request = require('request');

//Set whatever request options you like, see: https://github.com/request/request#requestoptions-callback
var requestArray = [
    {url: 'https://httpbin.org/get'},
    {url: 'https://httpbin.org/ip'}
];

let getApi = function (opt, callback) {
    request(opt, (err, response, body) => {
        callback(err, JSON.parse(body));
    });
};

const functionArray = requestArray.map((opt) => { 
    return (callback) => getApi(opt, callback); 
});

async.parallel(
    functionArray, (err, results) => {
        if (err) {
            console.error('Error: ', err);
        } else {
            console.log('Results: ', results.length, results);
        }
});

You can easily switch the Url and Query values to match whatever you need. I'm using HttpBin here, since it's good for illustrative purposes.

edited Aug 9, 2018 at 16:53

answered Aug 9, 2018 at 16:43

Terry Lennox

30.9k5 gold badges37 silver badges45 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Terry Lennox Over a year ago

That should be fine, you'd just form the Url for each request based on the arguments passed. I'll update my answer!

Terry Lennox Over a year ago

I've modified the answer so the requests are now in an array, you can set Url, Query string, headers for each one. In this case there are only two, however, you can create or map it as you wish.

ang Over a year ago

Thank you for this example! It's going to take me a minute to wrap my head around it. I'm running into problems calling the same API endpoint consecutively the JSON parsing fails...

Terry Lennox Over a year ago

You don't have to include the JSON.parse, you can just pass the body if you like, e.g. callback(err, body). This will pass a string rather than an object.

ang Over a year ago

I see, so perhaps I should parse the body in the API controller?

|

Collectives™ on Stack Overflow

How to Run an API Calls in Parallel (Node.js)

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related