2

I have a script that uses a selenium webdriver (geckodriver) and loads various webpages. The scripts works fine at the beginning, but than at a random point it stops working without raising any error (the program sorts of hangs without really doing anything).

I added some logging statement to check when it hangs, and this is caused by the WebDriverWait statement (see below). The last thing that is printed in the log is "get_records - Loaded".

The expected behavior to me would be to either print "get_records - Acquired pager", or to raise a TimeoutException after 10 seconds.

[...]
logging.info("get_records - Getting url: {}".format(url))
driver.get(url)
logging.info("get_records - Loaded")
# Get records number and result pages
elem = WebDriverWait(driver, 10).until(ec.element_to_be_clickable(
    (By.XPATH, "//td[@align='right']/span[@class='pager']"))
)
logging.info("get_records - Acquired pager")
[...]

Python version: 3.7.3 Selenium version: 3.141.0 Firefox version: 70.0.1

It seems like a similar bug happened with previous version (Selenium WebDriver (2.25) Timeout Not Working), but that bug was closed.

Is anyone having the same issue?

Update:

It seems like adding time.sleep(0.5) before elem prevents the script from freezing (either "get_records - Acquired pager" is printed, or the timeoutException is raised). Even though this is a turnaround for the issue, I would rather not put any forced wait.

Improve this question

1 Answer 1

Reset to default
5

I actually have the exactly same experience when the script works fine at first but hangs forever after some time. The '10 seconds' timeout is that webdriver/browser tries to open a page in 10 seconds. But the timeout that python script sends request to webdriver/browser is not defined. And it's none by default meaning request will wait infinitely.

Short answer:

driver.command_executor.set_timeout(10)
driver.get(url)

Explain:

Chromedriver as example. Whenever you run a selenium script. A process named 'chromedriver' starts as well. Let's call it 'control process'. It opens the browser and controls it. And it also acts as a http server which you can get the address and port by driver.command_executor._url. It receives http request, processes it, tells the browser to do something(maybe open a url) and returns. Details here.

When you call

elem = WebDriverWait(driver, 10).until(ec.element_to_be_clickable(
    (By.XPATH, "//td[@align='right']/span[@class='pager']"))
)

you are actually sending a request to the 'control process' which is a http server and tell it to do something(find some elements in current page). The timeout '10' means that 'control process' tells browser to open a page in 10 seconds before it cancels and returns timeout status to the python script.

But what really happens here is the 'control process' is receiving request but not responding. I don't really know what's happening in the 'control process'.

Python selenium package is using urllib3.request to send request and socket._GLOBAL_DEFAULT_TIMEOUT as timeout. It is none by default that makes a request wait infinitely. So you can set it by using driver.command_executor.set_timeout(10). Now if 'control process' brokes you will get a timeout exception and maybe recreate webdriver.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.