Python Selenium. How to use driver.set_page_load_timeout() properly?

Question

from selenium import webdriver

driver = webdriver.Chrome()
driver.set_page_load_timeout(7)

def urlOpen(url):
    try:
        driver.get(url)
        print driver.current_url
    except:
        return

Then I have URL lists and call above methods.

if __name__ == "__main__":
    urls = ['http://motahari.ir/', 'http://facebook.com', 'http://google.com']
    # It doesn't print anything

    # urls = ['http://facebook.com', 'http://google.com', 'http://motahari.ir/'] 
    # This prints https://www.facebook.com/ https://www.google.co.kr/?gfe_rd=cr&dcr=0&ei=3bfdWdzWAYvR8geelrqQAw&gws_rd=ssl

    for url in urls:
        urlOpen(url)

The problem is when website 'http://motahari.ir/' throws Timeout Exception, websites 'http://facebook.com' and 'http://google.com' always throw Timeout Exception.

Browser keeps waiting for 'motahari.ir/' to load. But the loop just goes on (It doesn't open 'facebook.com' but wait for 'motahari.ir/') and keep throwing timeout exception

Initializing a webdriver instance takes long, so I pulled that out of the method and I think that caused the problem. Then, should I always reinitialize webdriver instance whenever there's a timeout exception? And How? (Since I initialized driver outside of the function, I can't reinitialize it in except)

Why are you trying to load a webpage in under 5 seconds....that’ll be difficult... — Taku
– Taku, Commented Oct 11, 2017 at 5:22
I don’t think it should happen though. Everything seems correct, (other than the fact that you hadn’t import By yet) it might be just because your internet is slow or the website’s server is slow... — Taku
– Taku, Commented Oct 11, 2017 at 5:39
"But the loop just goes on (It doesn't open 'b.net' but wait for 'a.com') and keep throwing timeout exception" It doesn't, it actually is loading b.net, it just doesn't show it on the top. And I'll like you to know that links in selenium requires a prefix "https://" or "http://" to work. — Taku
– Taku, Commented Oct 11, 2017 at 6:03
Sorry, I had a method to add http but I omitted that for simplicity. I changed the code a little bit. Could you have a look at it, please? — Sean
– Sean, Commented Oct 11, 2017 at 6:17

Taku · Accepted Answer · 2017-10-11 06:46:47Z

9

You will just need to clear the browser's cookies before continuing. (Sorry, I missed seeing this in your previous code)

from selenium import webdriver

driver = webdriver.Chrome()
driver.set_page_load_timeout(7)

def urlOpen(url):
    try:
        driver.get(url)
        print(driver.current_url)
    except:
        driver.delete_all_cookies()
        print("Failed")
        return

urls = ['http://motahari.ir/', 'https://facebook.com', 'https://google.com']

for url in urls:
    urlOpen(url)

Output:

Failed
https://www.facebook.com/
https://www.google.com/?gfe_rd=cr&dcr=0&ei=o73dWfnsO-vs8wfc5pZI

P.S. It is not very wise to do try...except... without a clear Exception type, is this might mask different unexpected errors.

answered Oct 11, 2017 at 6:46

Taku

34.1k12 gold badges79 silver badges88 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Sean Over a year ago

Sometimes it throws another timeout exception while deleting cookies... any idea?

Taku Over a year ago

No sorry, it never raised any errors on mine, so I don’t know why it happened on yours D:

Gregor Leban Over a year ago

when I set set_page_load_timeout() and later call delete_all_cookies() I get a timeout exception. Why would delete_all_cookies() be throwing a timeout exception?

Collectives™ on Stack Overflow

Python Selenium. How to use driver.set_page_load_timeout() properly?

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related