python selenium wait for page to load

Question

I've written a script that gets data from a page, but sometimes the page takes time to load and so when it pull the html into a soup object sometimes It pulls nothing as the page still needs to finish.

I wrote the following code for waiting the page to finish.

def scrape_page(url):
     browser.get(url)    
     try:
        WebDriverWait(browser, 10).until(EC.presence_of_element_located(browser.find_element_by_id ("selection-box")))
        #Extract Source Code 
        html = browser.page_source;
        soup = BeautifulSoup(html)

It works

But I'm getting the following error when I call the function;

TypeError: find_element() argument after * must be a sequence, not WebElement

WKPlus · Accepted Answer · 2017-05-11 06:50:56Z

14

I think you should use presence_of_element_located like this:

element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "myDynamicElement"))
    )

as described in the manual.

edited May 11, 2017 at 6:50

answered May 20, 2015 at 3:45

WKPlus

7,3152 gold badges38 silver badges55 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Grant McKinnon Over a year ago

I tried this originallly, , but get the error global name By is not defined. Only thing that is different is that Im using Chrome Driver instead of firefox

sobolevn Over a year ago

selenium-python.readthedocs.org/en/latest/…

WKPlus Over a year ago

@GrantMcKinnonG import By before use it: from selenium.webdriver.common.by import By

Learner Over a year ago

@GrantMcKinnon- You have to import as from selenium.webdriver.common.by import By

Andrew Over a year ago

The link in this answer has died. RIP link.

B. Okba · Accepted Answer · 2022-03-13 08:09:36Z

5

After 6 years from this question, all the solutions I looked for was not good for my situation, I found the solution in other programming language and I did implementation for Python, so this is the best way to wait a page to fully load.

WebDriverWait(self.driver, self.SERVER_TIMEOUT).until(
        lambda wd: self.driver.execute_script("return document.readyState") == 'complete',
        "Page taking too long to load"
    )

answered Mar 13, 2022 at 8:09

B. Okba

1,2321 gold badge15 silver badges16 bronze badges

Comments

Dharman · Accepted Answer · 2021-04-20 14:38:42Z

2

I apply this func to every WebElement I need to use.

from selenium import webdriver

def FindElem(Driver: webdriver, XPath: str, Timeout: int = 300):
    while Timeout > 0:
        try:
            return Driver.find_element_by_xpath(XPath)
        except: # if element isn't already loaded or doesn't exist
            time.sleep(1)
            Timeout -= 1
    raise RuntimeError(f"Page loading timeout") # or whatever the hell you want

Usage:

Driver = webdriver.Firefox()
webdriver.get("http://somewhere.com/somepage.html")
MyWebElement = FindElem(Driver, "//input[@name='email']") # raise exception if timeout

edited Apr 20, 2021 at 14:38

Dharman♦

33.9k27 gold badges106 silver badges157 bronze badges

answered Apr 20, 2021 at 14:32

Emil Valeev

211 bronze badge

Comments

phpjunkie · Accepted Answer · 2024-06-11 12:43:54Z

Load the class for the browser your using and use implicitly_wait. implicitly_wait will wait for whatever element your trying to find. This is built into selenium and works perfectly for finding WebElements on a page that hasn't finished loading.

from datetime import datetime

from selenium.webdriver import Firefox
from selenium.webdriver.common.by import By
from selenium.common.exceptions import NoSuchElementException

firefox = Firefox()
firefox.implicitly_wait(time_to_wait = 120)  # in seconds: 2 minutes

firefox.get('https://www.google.com/')

timestamp = datetime.now().timestamp()
try:
    firefox.find_element(by = By.LINK_TEXT, value = 'aedgsf')
except NoSuchElementException:
    pass
time = datetime.fromtimestamp(datetime.now().timestamp() - timestamp).strftime('%M:%S:%f')
print(time)

>>> 02:00:910277

Collectives™ on Stack Overflow

python selenium wait for page to load

4 Answers 4

5 Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

5 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related