Nested for loop combined with fetching data from an API in Nodejs?

Question

My scenario is as follows: There is an API I would like to fetch from. The API returns a json that has an array named "assets". This array size will always be 20. Now, I call the endpoint like so:

 fetch(
     `https://api.example.io/api/v1/assets/?offset=${offset}`
 )

where if offset is 0 it will return the array of the 0 - 20 assets, if the offset is 20 it will return 20 to 40 and so on.

I want to check 1000 items which means I would like to call this fetch 1000/20 = 50 times. Whenever I call the fetch I would like loop over these 20 items and insert them into my DB. The problem is that I can't do something like this:

let offset=0;

for(let i = 0; i < 50; i++ {
    fetch(
       `https://api.example.io/api/v1/assets/?offset=${offset}`
    )
    for(let j = 0; j < 20; j++){
    // Insert into DB
    }
    offset+=20;
}

Due to JS asynchronous nature. Whenever I try to do this it will call the fetch with the value 0 for offset a couple of times, it won't wait until the nested for loop finishes and than call it for 20 and later 40 and so on...

What is the correct way to achive this behavior?

Andy · Accepted Answer · 2018-12-03 19:03:59Z

1

I've nicked the following from one of my nodeJS repos as it employs async code for solving a very similar problem:

// Throttling is important because you don't want to
// overload the API
const createThrottle = require('async-throttle');
const throttle = createThrottle(2);

// First push all the links into an array using
// the offset
const links = [];
for (let offset = 0; offset < 100; offset += 20) {
  links.push(`https://api.example.io/api/v1/assets/?offset=${offset}`);
}

// Next create an array of promises by `map`ing
// over the links using `fetch`.
// Notice I've used throttle here to slow down the hits on
// the API
const promises = links.map(link => throttle(async => () {
  const res = await fetch(link);
  return await res.json();
}));

// Once all the promises are complete, iterate over their datasets
Promise.all(promises).then(datasets => {
  // iterate over datasets
});

edited Dec 3, 2018 at 19:03

answered Dec 3, 2018 at 18:53

Andy

63.6k13 gold badges72 silver badges99 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

mwilson · Accepted Answer · 2018-12-03 18:47:55Z

Why not use Promise.all?

const promises = [];

for (let i = 0; i < 10; i++) {
    promises.push(
    fetch(`https://api.example.io/api/v1/assets/?offset=${i}`)
  );
}

Promise.all(promises).then( _results => {
    // All fetch calls have completed at this point. _results will be in array of your fetch results.
    console.log(results);
    // Do db insert here

});

One thing you could do is to make a function that is a promise. Within that function, have it execute the fetch and then execute the db insert all in one function (using .then). If you did it this way, the single Promise.all call would handle everything. If you don't, you'll have to loop through the promises again and insert those values into the db. It might look something like this:

const promises = [];

for (let i = 0; i < 10; i++) {
    promises.push(fetchAndInsert(i));
}

Promise.all(promises).then( _results => {
    console.log(results);

});


function fetchAndInsert(offset) {
    return new Promise( (resolve, reject) => {
    fetch(`https://api.example.io/api/v1/assets/?offset=${i}`).then (_result => {
        // db.insert().then( _result => {
            //resolve();
      //})
    })
  })
}

weibenfalk · Accepted Answer · 2018-12-03 18:49:56Z

0

You could use async and await. This should work:

async function fetchExample() {
  for (let i = 0; i < 50; i++) {
    let fetchedData = await fetch(
      `https://api.example.io/api/v1/assets/?offset=${offset}`
      );
    for(data of fetchedData) {
      // Insert into DB
    }
  offset+=20;
  }
}

answered Dec 3, 2018 at 18:49

weibenfalk

9027 silver badges10 bronze badges

Comments

Satpal Tanan · Accepted Answer · 2018-12-03 18:54:45Z

Instead of for..loop, you could use recursion or Promises.

Recursion::

let insertToDB = function (records, cb) {
  if (!records.length) return cb();
  let record = records.shift();//assuming records is an array
  // Insert record into DB1
  insertToDB(records, cb);
};

let loop = function (count, offset, cb) {
 if (!count) return cb();
 fetch(
    `https://api.example.io/api/v1/assets/?offset=${offset}`
 )
 insertToDB(recordsFromFetch, function () {
    offset += 20;
    --count;
    loop(count, offset, cb)
 })
};


loop(50, 0, function () {
  console.log("All Done");
})

Promise:: I have not run it, so might be some syntactical error, but you get the idea

let insertToDB = function (record) {
    return new Promise(function (resolve, reject) {
    // Insert record into DB then call resolve
    resolve();
    })
};

let fetchPhase = function (offset) {
    fetch(
    `https://api.example.io/api/v1/assets/?offset=${offset}`
   )
    let dbAll = [];
   for (let j = 0; j < 20; j++) {
    // assuming you get records from fetch , pass record to be added to db in the parameter
    dbAll.push(insertToDB(records[j]))
}
 return Promise.all(dbAll)
};

let all = [];

let offset = 0;

for (let i = 0; i < 50; i++) {
    all.push(fetchPhase(i));
}

Promise.all(all)
.then(function () {
    console.log("ALL DONe");
})

rich remer · Accepted Answer · 2018-12-03 23:34:54Z

0

Promise.all(Array(50).fill(null).map((v,i) => {
    const url = `https://api.example.io/api/v1/assets/?offset=${i*20}`;

    return fetch(url).then(results => {
        for (let result of results) {
            // insert into DB
        }
    );
}).then(() => console.log("done"));

answered Dec 3, 2018 at 23:34

rich remer

3,6162 gold badges38 silver badges49 bronze badges

Collectives™ on Stack Overflow

Nested for loop combined with fetching data from an API in Nodejs?

5 Answers 5

Comments

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related