Javascript - Infinite scroll loop

Question

Short story :

I want to parse a website that has infinite scroll on it, I don't want to implement infinite scroll. So I want to create a script that automatically scroll the page from the browser console wait for the data to appears then repeat until the end.

Long story :

I'm trying to scroll down an infinite scroll using javascript console into my browser. When we scroll to the bottom, an Ajax call is made and fill a <div> element so we have to wait for this to occurs then resume our process.

My main problem here is that all of this is done too quickly, it doesn't wait the ajax to be completed before resuming the process.

To have a concrete example, when I go to AngelList jobs page (need to be logged in) and when I put this in my browser console :

function sleep(µs) {
     var end = performance.now() + µs/1000;
     while (end > performance.now()) ; // do nothing
}

var loading = true;

$(window).load(function () {
   loading = false;
}).ajaxStart(function () {
   loading = true;
}).ajaxComplete(function () {
   loading = false;
});

function waitAfterScroll() {
    if(loading === true) {
        console.log("Wait the Ajax to finish loading");
        setTimeout(waitAfterScroll, 1000);
        return;
    }

    console.log("Ajax DONE");
}

var X= 0;
while(X < 100){
  sleep(1000000);
  X = X + 10;
  console.log(X);
  window.scrollTo(0,document.body.scrollHeight);

  waitAfterScroll();
}

I've got this result :

10
Wait the Ajax to finish loading
20
Wait the Ajax to finish loading
30
Wait the Ajax to finish loading
40
Wait the Ajax to finish loading
50
Wait the Ajax to finish loading
60
Wait the Ajax to finish loading
70
Wait the Ajax to finish loading
80
Wait the Ajax to finish loading
90
Wait the Ajax to finish loading
100
Wait the Ajax to finish loading
Wait the Ajax to finish loading
Ajax DONE

What I would like is :

10
Wait the Ajax to finish loading // one or multiple times.....
Ajax DONE
20
Wait the Ajax to finish loading // one or multiple times.....
Ajax DONE
30
Wait the Ajax to finish loading // one or multiple times.....
Ajax DONE
40
Wait the Ajax to finish loading // one or multiple times.....
Ajax DONE
50
Wait the Ajax to finish loading // one or multiple times.....
Ajax DONE
60
Wait the Ajax to finish loading // one or multiple times.....
Ajax DONE
70
Wait the Ajax to finish loading // one or multiple times.....
Ajax DONE
80
Wait the Ajax to finish loading // one or multiple times.....
Ajax DONE
90
Wait the Ajax to finish loading // one or multiple times.....
Ajax DONE
100
Wait the Ajax to finish loading // one or multiple times.....
Ajax DONE

I hope this is clear.

In other words, I would like to be able to scroll down, stop the execution of the javascript or at least wait for the ajax to finish loading, then repeat.

A quick Google search brings up many examples. Here's one phppot.com/jquery/… — Darkrum
– Darkrum, Commented Sep 8, 2016 at 23:25
Thanks @AlexFilatov but not exactly. I want to parse a website that has infinite scroll on it, I don't want to implement infinite scroll. So I want to automatically scroll the page via a script. — SJU
– SJU, Commented Sep 9, 2016 at 7:33

Federico · Accepted Answer · 2016-09-09 00:19:20Z

3

I am pretty sure that I misunderstood what you wanted, but anyway. Why not using Promises?:

var promise = new Promise(function(resolve, reject) {
  var req = new XMLHttpRequest();
  req.open('GET', url);

  req.onload = function() {
    if (req.status == 200) {
       resolve(req.response);
  }
};
});

promise.then(function(result) {
  window.scrollTo(0,document.body.scrollHeight);
})

Or, you could make a synchronous ajax request to stop javascript execution.

 jQuery.ajax({
    url: "someurl.html",
    async: false, 
    success: function(html){ 
      window.scrollTo(0,document.body.scrollHeight);
    }
  });

But, for performance reasons, I would not recommend that. Instead you could:

// Start off with a promise that always resolves
var sequence = Promise.resolve();
arrayOfWebsiteContentUrls.forEach(function(url) {
  sequence = sequence.then(function() {
    return fetchTheContent(url);
  }).then(function(content) {
    addContent(content);                 
    window.scrollTo(0,document.body.scrollHeight);
  });
});

I took that last bit from this excelent article on Promises.

Please, let me know if there is something that you don't get.

edited Sep 9, 2016 at 0:19

answered Sep 8, 2016 at 23:47

Federico

4,0005 gold badges38 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

SJU Over a year ago

Thanks for your answer. I've edited my post because I think I was not clear previously. I want to parse a website that has infinite scroll on it, I don't want to implement infinite scroll. So I want to create a script that automatically scroll the page from the browser console wait for the data to appears then repeat until the end.

Federico Over a year ago

@StéphaneJ Ah, I get it now, but do you have to display the site as is (with css and all) or you just want to scrape its content? If that's the case, you could mimic the ajax calls the site makes. Let me know if that's the case to elaborate it further.

Collectives™ on Stack Overflow

Javascript - Infinite scroll loop

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related