Explicit Wait not timing out consistently in Selenium Python

Question

I have the following code using Selenium in Python 3:

profile = webdriver.FirefoxProfile()
profile.set_preference('webdriver.load.strategy', 'unstable')
browser = webdriver.Firefox(profile)
browser.set_page_load_timeout(10)
url = 'my_url'
while True:
    try:  
        st = time.time()
        browser.get(url)
        print('Finished get!')
        time.sleep(2)
        wait = WebDriverWait(browser, 10)
        element = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'div[my_attr="my_attr"]')))  
        print('Success after {} seconds.'.format(round(time.time()-st)))  
        break
    except:
        print('Timed out after {} seconds.'.format(round(time.time()-st)))
        print('Reloading')
        continue

From my understanding, using the explicit wait here (even with the unstable load strategy and page load timeout), what should happen is that the page should load, it should look for the element specified, and if either the page doesn't load within 10 seconds or the element is not found within 10ish seconds, it should time out and reload again (because of the try/except clause with the while loop).

However, what I'm finding is that it's not timing out consistently. For example, I've had instances where the loading times out after 10ish seconds the first time around, but once it reloads, it doesn't time out and instead "succeeds" after like 140 seconds. Or sometimes it doesn't time out at all and just keeps running until it succeeds. Because of the unstable load strategy, I don't think the page load itself is ever timing out (more specifically, the 'Finished get!' message always prints). But the explicit wait here that I specified also does not seem to be consistent. Is there something in my code that is overriding the timeouts? I want the timeouts to be consistent such that if either the page doesn't load or the element isn't located within 10ish seconds, I want it to timeout and reload. I don't ever want it to go on for 100+ seconds, even if it succeeds.

Note that I'm using the unstable webdriver load strategy here because the page I'm going to takes forever to completely load so I want to go straight through the code once the elements I need are found without needing the entire page to finish loading.

Is this the entirety of the code? You don't have an implicit wait buried somewhere do you? Mixing explicit and implicit waits are known to cause weird issues like this. I would expect that your timeouts should range from 10-22s. 10s if the page load timeout is hit and 22s if the element isn't found... 10s for page load timeout + 2s for .sleep() + 10s for explicit wait. — JeffC
– JeffC, Commented Feb 25, 2017 at 14:48
You should remove the .sleep(). it's generally not a good practice to use it and it shouldn't be needed here anyway because you have an explicit wait right after it. The continue can be safely removed also. — JeffC
– JeffC, Commented Feb 25, 2017 at 14:51
I'm starting to wonder if the second .get() isn't interrupting the first if it's still running even though it throws an error. Can you see that the page reloads again after ~22 seconds? Maybe change the URL after the except and give it "cnn.com" or something to see if it will redirect while it's still trying to load. Even if that works, there still may be the issue where the browser won't try to load the page again if it's already trying to load it. I don't use python so I can't test any of this... sorry I can't be more help but this is an interesting problem. — JeffC
– JeffC, Commented Feb 25, 2017 at 15:03
Thanks for the tips. To answer your questions, there is additional code related to logging in to the website, but I was careful about not including any implicit waits. As far as I can tell, there is no additional code that should affect the waits. Also, changing the URL after the except doesn't seem to affect it. The waits aren't consistent to the extent that sometimes on the first loop (without ever raising an exception), the waits are ineffective. So it sometimes happens that the first iteration takes 100+ seconds and just never times out. — plam
– plam, Commented Feb 26, 2017 at 7:42

plam · Accepted Answer · 2017-02-26 09:59:17Z

1

After some more testing, I have located the source of the problem. It's not that the waits are not working. The problem is that all the time is being taken up by the locator. I discovered this by essentially writing my own wait function and using the .find_element_by_css_selector() method, which is where all the runtime is occurring when it takes 100+ seconds. Because of the nature of my locator and the complexity of the page source, it's taking 100+ seconds sometimes for the locator to find the element when the page is nearly fully loaded. The locator time is not factored into the wait time. I presume that the only "solution" to this is to write a more efficient locator.

answered Feb 26, 2017 at 9:59

plam

1,3853 gold badges17 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Explicit Wait not timing out consistently in Selenium Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related