While loop inside a .then() async

Question

I recently came across an interesting problem that I cannot solve for the life of me. I am calling an API call that is breaking up records into pages, the info for the pages reside in the response header. From that, I wanted to be able to do another call to retrieve the data and the next header, until there are no more response headers.

let parents = {};

const options = {
  credentials: "same-origin",
  headers: {
    accept: "application/json"
  },
  timeout: 5000
};
fetch(
  `/api/v1/courses/200003/enrollments?enrollment_type=ObserverEnrollment&per_page=100`,
  options
).then(response =>
  response
    .json()
    .then(data => ({
      data: data,
      ok: response.ok,
      headers: response.headers
    }))
    .then(res => {
      parents = res;

      nextURL(res.headers.get("Link"));

      let optionsNext = {
        credentials: "same-origin",
        headers: {
          accept: "application/json"
        },
        timeout: 5000
      };
      fetch(nextURL(res.headers.get("Link")), optionsNext).then(response =>
        response
          .json()
          .then(data => ({
            data: data,
            ok: response.ok,
            headers: response.headers
          }))
          .then(res => {
            if (res.ok) {
              parents.data.push(res.data);
              console.info(parents);
            }
          })
      );
    })
);

function nextURL(linkTxt) {
        let url = null;
        if (linkTxt) {
          let links = linkTxt.split(",");
          let nextRegEx = new RegExp('^<(.*)>; rel="next"$');

          for (let i = 0; i < links.length; i++) {
            let matches = nextRegEx.exec(links[i]);
            if (matches) {
              url = matches[1];
            }
          }
        }
        return url;
      }

The part that I need to put into some kind of loop is the secondary fetch based upon the return of the nextURL function: if !nextURL(res.headers.get("Link")) I need to break the loop.

let optionsNext = {
        credentials: "same-origin",
        headers: {
          accept: "application/json"
        },
        timeout: 5000
      };
      fetch(nextURL(res.headers.get("Link")), optionsNext).then(response =>
        response
          .json()
          .then(data => ({
            data: data,
            ok: response.ok,
            headers: response.headers
          }))
          .then(res => {
            if (res.ok) {
              parents.data.push(res.data);
              console.info(parents);
            }
          })
      );

Thanks in advance for even looking at my pitiful problem

You don't really need a while loop, you should use recursion. When the first fetch resolves, check if there's a next page, if there is, rerun the function. You really just need a dispatcher object so that you can call the fetch method recursively. — gabriel.hayes
– gabriel.hayes, Commented Jan 16, 2020 at 15:52
thank God. I could use a bit more hand holding though @user1538301. Could you please expound a bit more. — Jewell Simon
– Jewell Simon, Commented Jan 16, 2020 at 15:56
I'm attempting to put together an example, though it won't be a working one (don't know the API you're using lol), one moment. — gabriel.hayes
– gabriel.hayes, Commented Jan 16, 2020 at 15:57
TY, it is the Canvas API BTW, canvas.instructure.com/doc/api/file.pagination.html. — Jewell Simon
– Jewell Simon, Commented Jan 16, 2020 at 15:59

gabriel.hayes · Accepted Answer · 2020-01-16 18:53:33Z

2

Try using recursion; something like this:

const getFetcher = () => ({
    aggregate: [],
    run: function (url, options) {
        return new Promise((resolve, reject) => {

            fetch(url, options)
                .then(response => {
                    const json = response.json();
                    const { headers, data } = response;
                    const nextLink = res.headers.get("Link");
                    this.aggregate.push(data);
                    if (nextLink) {
                        this.run(nextLink, options).then(resolve);
                    }
                    else {
                        resolve(this.aggregate);
                    }
                })

        })
    }
})
const options = {
    credentials: "same-origin",
    headers: {
        accept: "application/json"
    },
    timeout: 5000
};
const fetcher = getFetcher();
fetcher.run(`/api/v1/courses/200003/enrollments?enrollment_type=ObserverEnrollment&per_page=100`, options)
    .then(allPagesData => {
        /* at this point you have all the pages data */
    })

edited Jan 16, 2020 at 18:53

answered Jan 16, 2020 at 16:06

gabriel.hayes

2,31312 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

gustafc Over a year ago

No need for messing about with new Promise - let run return the result of fetch(url, options).then(...), and let the callback to thenend with return nextLink ? this.run(nextLink) : this.aggregate. This way an unexpected exception within the then won't result in a forever-pending promise.

gabriel.hayes Over a year ago

@gustafc that sounds like it'll just result in a Promise being returned before all of the pages are fetched.

gabriel.hayes Over a year ago

I mean think about it... if you just return fetch, and it has a nextLink, you return this.run(nextLink) which is a Promise.. wouldn't fetcher.run().then get a Promise in that case..? Also the only way to avoid a forever-pending promise in this case is to add a counter of some sort, ending prematurely isn't a desired alternative to a never-ending loop (which would technically be a problem on the API side.. as that would only happen if their data had a circular reference)

gustafc Over a year ago

My proposition does the exact same thing your code did, but with less code - just a little suggestion for improvement. Of course it returns a promise before all pages are fetched - that's the whole idea with promises! But the promise doesn't resolve until an invocation of run got into the branch wherethis.aggregate is returned.

gustafc Over a year ago

About the forever-pending promise thingy, I mis-remembered how new Promise handles uncaught exceptions - my bad. Still, it's always simpler to not use new Promise unless one absolutely has to, which one only does when interfacing with callback-based APIs.

|

Razvan Tanase · Accepted Answer · 2020-01-16 16:54:40Z

1

Use async recursive functions.
I'm not exaclty sure what your api returns but this should work:
Firstly you can just return the element when you find it, it saves you a few iterations if there are too many of them.

function nextURL(linkTxt) {
    if (linkTxt) {
      let links = linkTxt.split(",");
      let nextRegEx = new RegExp('^<(.*)>; rel="next"$');

      for (let i = 0; i < links.length; i++) {
        let matches = nextRegEx.exec(links[i]);
        if (matches && matches[1]) {
            //return right away
            return matches[1];
        }
      }
    }
    return null;
}

Next define your main call:

const OPTIONS = {
  credentials: "same-origin",
  headers: {
    accept: "application/json"
  },
  timeout: 5000
};
let parents = {};
async function main(){
    const RESPONSE = await fetch(`/api/v1/courses/200003/enrollments?enrollment_type=ObserverEnrollment&per_page=100`,OPTIONS);
    let data = await RESPONSE.json();
    let res = {
        data: data,
        ok: RESPONSE.ok,
        headers: RESPONSE.headers
    }

    loop(res);
    //or: 
    //await loop(res);
    //if you want to wait for it.
}

And then your loop

const OPTIONS_NEXT = {
    credentials: "same-origin",
    headers: {
      accept: "application/json"
    },
    timeout: 5000
};
async function loop(parents){
    //if nextURL returns null...
    if(nextURL(parents.headers.get("Link")),OPTIONS_NEXT === null) 
        //...end the loop
        return;
    //otherwise keep going.

    const RESPONSE = await fetch(nextURL(parents.headers.get("Link")),OPTIONS_NEXT);
    let data = await RESPONSE.json();
    let res = {
        data: data,
        ok: RESPONSE.ok,
        headers: RESPONSE.headers
    }
    if (res.ok) {
        parents.data.push(res.data);
        console.info(parents);
    }
    loop(res);
    //or: 
    //await loop(res);
    //if you want to wait for it.
    //You need to call it from within an async function.
}

Now all you need to do is call the main function:

main();
//or: 
//await main();
//if you want to wait for it.
//You need to call it from within an async function.

answered Jan 16, 2020 at 16:54

Razvan Tanase

991 silver badge8 bronze badges

2 Comments

Razvan Tanase Over a year ago

Oh I see, I'm sorry about that. In that case you could change your nextURL function so that it returns false instead of null and test the explicit condition as if(nextURL(parents.headers.get("Link")),OPTIONS_NEXT === false). The reason you should test it as if(condition === false) is because if(!condition) actually tests true for false aswell as undefined and null.

Jewell Simon Over a year ago

Thank you @RazvanTanase. This answer is really awesome(∩_∩). This is pretty awesome as well. I tested it and it works, but the loop errors out which causes a break then leaves a hanging promise. if(nextURL(parents.headers.get("Link")),OPTIONS_NEXT === null) actually returns the api url/null changing it to if(!nextURL(parents.headers.get("Link"))) works great

Collectives™ on Stack Overflow

While loop inside a .then() async

2 Answers 2

9 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

9 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related