Scrolling an element which loads further on scrolling using Selenium in Python

Question

I am trying to create an Instagram scraping bot that collects a list of Followers and Following using Python + Selenium.

However, that list keeps on loading when the user scrolls until the list is exhausted. I am attaching a screenshot for reference (some content hidden due to privacy reasons):

Now, I believe I have two ways to achieve this:

Keep reading the usernames, and then keep on scrolling.
Keep scrolling till the end, and then read all usernames together from the source code.

I've been trying to figure this out using the second method. However, I am not able to figure out how to know when there is no more content to scroll. How can I achieve this (provided that I don't know anything about the length of this element)?

Reason for not using Method 1: When scrolling, the DOM keeps getting refreshed, so it is hard to keep track of which usernames have been read.

Wouter · Accepted Answer · 2021-01-26 16:54:07Z

1

One way to do this is is to keep track of the amount of child elements the div has that contains the li elements for the followers. If it doesn't increase after a scroll event, you've reached the end of the list.

answered Jan 26, 2021 at 16:54

Wouter

2854 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Priyanshul Govil Over a year ago

Yes, this crossed my mind. But this would require to check the count of li elements after every scroll. That seemed to be quite a waste of computational power.

Priyanshul Govil Over a year ago

I tried some other things as well like scrolling down total_users / users_scrolled_in_one_KEY_DOWN times but that did not work smoothly. So I adapted your idea but changed KEY_DOWN to END signal so there are fewer repetition in calculating the count of li tags. Leaving it out in the comments as it might help someone else.

Priyanshul Govil Over a year ago

I was expecting something which could provide me with a signal when scrolling can't be done further. However, this has helped. Accepting this solution for now.

Collectives™ on Stack Overflow

Scrolling an element which loads further on scrolling using Selenium in Python

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related