The web scraping issue in python, web page not loading in time

Question

I am making a program for scrapping the Amazon websites mobile phones but my program is giving me timeout exception even after the page is loaded on time.
Here is my code

    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.common.by import By
    from selenium.common.exceptions import TimeoutException
    from selenium.common.exceptions import NoSuchElementException

    from bs4 import BeautifulSoup
    import urllib.request

    class Amazon_all_mobile_scraper:
        def __init__(self):
            self.driver = webdriver.Firefox()
            self.delay = 60
            self.url = "https://www.amazon.in/mobile-phones/b/ref=sd_allcat_sbc_mobcomp_all_mobiles?ie=UTF8&node=1389401031"

        def load_amazon(self):
            self.driver.get(self.url)
            try:
                wait = WebDriverWait(self.driver,self.delay)
                        wait.until(EC.presence_of_element_located((By.CLASS_NAME,"acs-ln-link")))
                print("Page is ready.")
            except TimeoutException:
                print("Took too much time to load!")
            except:
                print("Something went wrong in loading part!!")

        def extract_list_of_mobiles(self):
            try:
                mobile_list = self.driver.find_element_by_xpath('//div[@class = "acs-ln-link"]')
                print(mobile_list)
            except NoSuchElementException:
                print("Sorry, Unable to get the requested element")


    scraper = Amazon_all_mobile_scraper()
    scraper.load_amazon()
    scraper.extract_list_of_mobiles()

Please help me to figure out whats wrong in this code.

What is it you wanna grab from there? Your xpath matches nothing. — SIM
– SIM, Commented Oct 31, 2018 at 7:00
That class name is not present on the page as far as I can see. — QHarr
– QHarr, Commented Oct 31, 2018 at 7:23

SIM · Accepted Answer · 2018-10-31 08:21:51Z

1

Only changing from acs-ln-link to acs-ln-links will not do the trick. Your xpath should look more like '//div[contains(@class,"acs-ln-nav-expanded")]//*[@class="acs-ln-links"]//a'. This is, however, you can cope with to get the required output:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

class Amazon_all_mobile_scraper:

    url = "https://www.amazon.in/mobile-phones/b/ref=sd_allcat_sbc_mobcomp_all_mobiles?ie=UTF8&node=1389401031"

    def __init__(self):
        self.driver = webdriver.Chrome()
        self.wait = WebDriverWait(self.driver, 15)

    def load_n_get_from_amazon(self):
        self.driver.get(self.url)
        mobile_list = self.wait.until(EC.presence_of_all_elements_located((By.XPATH,'//div[contains(@class,"acs-ln-nav-expanded")]//*[@class="acs-ln-links"]//a')))
        return mobile_list

    def __del__(self):
        self.driver.close()

if __name__ == '__main__':
    scraper = Amazon_all_mobile_scraper()
    for item in scraper.load_n_get_from_amazon():
        print(f'{item.text}\n{item.get_attribute("href")}\n')

answered Oct 31, 2018 at 8:21

SIM

22.5k6 gold badges45 silver badges116 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

SIM Over a year ago

You should change self.driver = webdriver.Chrome() to self.driver = webdriver.Firefox() for the script to get going.

Star Rider Over a year ago

Could you please clarify how you formed that XPath or suggest some website from which I can learn to make one that you have written.

SIM Over a year ago

That is beyond the scope of your original question @Star Rider. However, you can check out this link to learn how xpath can be formed. Thanks.

Star Rider · Accepted Answer · 2018-10-31 08:13:47Z

1

The class wasn't matching "acs-ln-link" should be "acs-ln-links".

answered Oct 31, 2018 at 8:13

Star Rider

3893 silver badges15 bronze badges

Collectives™ on Stack Overflow

The web scraping issue in python, web page not loading in time

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related