0

By chrome web loading scroll I mean the dynamic loading icon at any chrome tab.

I need a kind of "WebDriverWait" to return an error if the loading icon from chrome doesn't stop scrolling until T seconds. Using pure WebDriverWait for checking the presence of some element would not be worth for me as WebDriverWait would just run after scrolling is done and I need to check if the selenium commands are already running (that's when scrolling is done). I tried to do this by using the method set_page_load_timeout:

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.service import Service
options = webdriver.ChromeOptions()
options.add_argument("--start-maximized")
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), chrome_options=options)
driver.get('https://apps.varejonline.com.br/relatorios/relatorios/base/analisevendasperiodo/0')
driver.set_page_load_timeout(2) #I've set a time of 2 seconds just to test this methods

But this pages cleary takes (loading scroll is done) more than 2 seconds to load so driver.set_page_load_timeout(2) would supposed to me return an error but it didn't. Is this method outdated? How would I solve this problem?

1 Answer 1

0

From your code, you are loading the page before setting the page loading timeout hence a timeout exception will not be thrown after the 2 seconds

What you want is: set the page loading time before loading the page and execute the loading in a try...except statement to catch when the loading is timed out

def load_page(url: str, timeout: int = 20):
    try:
        driver.set_page_load_timeout(timeout)
        driver.get(url)
        return 'Loading Page Successful'
    except TimeoutException:
        return 'Loading Page Failed'

Alternatively, you could change the PageLoadStrategy of the driver. There are three strategies: normal, eager and none. Default is normal. This holds off all other driver activities until the page is ready. From my experience, I prefer using the eager strategy which then allows me to use WebDriverWait to check for readiness of elements on the page

Change the loadstrategy via the chrome options

options.set_capability('pageLoadStrategy', 'eager')

Then load your page and use WebDriverWait to check the elements on the page

Complete code method 1:

from selenium.common.exceptions import TimeoutException
from selenium.webdriver import Chrome
from selenium.webdriver.chrome.options import Options as ChromeOptions
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager


def load_page(driver: Chrome, url: str, timeout: int = 20):
    try:
        driver.set_page_load_timeout(timeout)
        driver.get(url)
        return 'Loading Page Successful'
    except TimeoutException:
        return ''
    finally:
        # reset the page load time, not necessary
        driver.set_page_load_timeout(1) # use default


if __name__ == '__main__':
    options = ChromeOptions()
    options.add_argument("--start-maximized")
    driver = Chrome(service=Service(ChromeDriverManager().install()), options=options)

    url = 'https://apps.varejonline.com.br/relatorios/relatorios/base/analisevendasperiodo/0' # this url is not responding
    loaded = load_page(driver, url, 60)
    if not loaded:
        print(f'the page {url} is taking too long to load')
        # quit or do something else
    else:
        print('Page loaded')
        # search your elements, continue crawling

    driver.quit()

Complete code Method 2:

from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver import Chrome
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options as ChromeOptions
from selenium.webdriver.chrome.service import Service
from selenium.common.exceptions import TimeoutException
from webdriver_manager.chrome import ChromeDriverManager


def find_element(driver: Chrome, locator: tuple[str, str], timeout: int = 40):
    try:
        element = WebDriverWait(driver, timeout=timeout).until(EC.presence_of_element_located(locator))
        return element
    except TimeoutException:
        return None


if __name__ == '__main__':
    options = ChromeOptions()
    options.add_argument("--start-maximized")
    options.set_capability('pageLoadStrategy', 'eager')

    driver = Chrome(service=Service(ChromeDriverManager().install()), options=options)

    url = 'https://apps.varejonline.com.br/relatorios/relatorios/base/analisevendasperiodo/0' # this url is not responding
    driver.get(url)
    element = find_element(driver, (By.XPATH, '/html/path/to/element'), 30)
    if not element:
        print('Element not found')
    else:
        print(element.get_attribute('innerText'))

    driver.quit()
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.