How to move to the next page on Python Selenium?

Question

I am trying to build a proxy scraper for a specific site, but I'm failing on move to next page.

This is the code that I'm using.

If you answer my question, please, explain me a bit about what you used and if you can, please, if there are any good tutorials for over this kind of code, provide me some:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time

options = Options()
#options.headless = True     #for headless
#options.add_argument('--disable-gpu') #for headless and os win

driver = webdriver.Chrome(options=options)

driver.get("https://hidemyna.me/en/proxy-list/")
time.sleep(10) #bypass cloudflare


tbody = driver.find_element_by_tag_name("tbody")
cell = tbody.find_elements_by_tag_name("tr")

for column in cell:
    column = column.text.split(" ")
    print (column[0]+":"+ column[1]) #ip and port

nxt = driver.find_element_by_class_name('arrow_right')
nxt.click()

Try nxt = driver.find_element_by_css_selector('.arrow__right>a'). Note that there are two underscores in class name — Andersson
– Andersson, Commented Dec 28, 2018 at 9:12

undetected Selenium · Accepted Answer · 2018-12-29 09:14:52Z

5

To move to the next page you can try the following solution:

Code Block:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException, WebDriverException

options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get('https://hidemyna.me/en/proxy-list/')
while True:
    try:
        driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//li[@class='arrow__right']/a"))))
        driver.find_element_by_xpath("//li[@class='arrow__right']/a").click()
        print("Navigating to Next Page")
    except (TimeoutException, WebDriverException) as e:
        print("Last page reached")
        break
driver.quit()

Console Output:

Navigating to Next Page
Navigating to Next Page
Navigating to Next Page
Navigating to Next Page
Navigating to Next Page
Navigating to Next Page
Navigating to Next Page
Navigating to Next Page
Navigating to Next Page
Navigating to Next Page
Navigating to Next Page
Navigating to Next Page
Navigating to Next Page
Navigating to Next Page
Navigating to Next Page
Navigating to Next Page
Navigating to Next Page
Navigating to Next Page
Navigating to Next Page
Navigating to Next Page
.
.
.
Navigating to Next Page
Last page reached

edited Dec 29, 2018 at 9:14

answered Dec 28, 2018 at 9:18

undetected Selenium

194k44 gold badges304 silver badges387 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

undetected Selenium Over a year ago

@CodeIt Can you explain why this will not work on page 2?

Andersson Over a year ago

@DebanjanB , on second page your selector will match Previous button instead of Next

Haunter Over a year ago

tried the above but it is not working on this site. the error output is selenium.common.exceptions.WebDriverException: Message: unknown error: Element <a href="/en/proxy-list/?start=64#list"></a> is not clickable at point (874, 563). Other element would receive the click: <jdiv class="hoverl_6R"></jdiv>

undetected Selenium Over a year ago

@Haunter Checkout my answer update and let me know the status.

CodeIt · Accepted Answer · 2018-12-28 09:17:10Z

0

You are not actually clicking the anchor <a> tag. To navigate to next page you need to click on the <a> link.

You can use find_element_by_xpath like below.

driver.find_element_by_xpath('//*[@id="content-section"]/section[1]/div/div[4]/ul/li[1]/a').click()

Instead of using xpath you may use css selector as suggested by another @Andersson.

edited Dec 28, 2018 at 9:17

answered Dec 28, 2018 at 9:11

CodeIt

3,6283 gold badges28 silver badges37 bronze badges

Comments

Ton · Accepted Answer · 2020-01-10 19:14:59Z

0

the next button tend to vary from webpage to webpage...you will have to inspect the button and address it with xpath or beaufifulsoup

There's usually 'next page' and 'previous page'...address your xpath to 'next'

answered Jan 10, 2020 at 19:14

Ton

331 silver badge4 bronze badges

Collectives™ on Stack Overflow

How to move to the next page on Python Selenium?

3 Answers 3

4 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related