StaleElementReferenceException issue while scraping with Selenium

Question

I'm trying to load this page in full: https://candidat.pole-emploi.fr/offres/emploi/horticulteur/s1m1

I've set a line of code to handle the cookies popup.

Then I've set some lines to click on the Load More Results button in order to have the full html loaded and then printing it.

But I hit an error message after it clicked once :

StaleElementReferenceException: stale element reference: element is not attached to the page document

I don't know what it means nor how to fix it

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
import time

options = webdriver.ChromeOptions()
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')

site = 'https://candidat.pole-emploi.fr/offres/emploi/horticulteur/s1m1'
wd = webdriver.Chrome("C:\Program Files (x86)\chromedriver.exe", options=options)
wd.get(site)

time.sleep(10)

wait = WebDriverWait(wd, 10)

# click cookies popup
wd.find_element_by_xpath('//*[(@id = "description")]//*[contains(concat( " ", @class, " " ), concat( " ", "tc-open-privacy-center", " " ))]').click()

time.sleep(10)

# click show more button until no more results to load
while True:
    try:
        more_button = wait.until(EC.visibility_of_element_located((By.LINK_TEXT, 'AFFICHER LES 20 OFFRES SUIVANTES'))).click()
    except TimeoutException:
        break

time.sleep(10)

print(wd.page_source)
print("Complete")

time.sleep(10)
wd.quit()

use execute_script() i think that is more realiable

user15980977
– user15980977

2021-10-18 14:20:54 +00:00
Commented Oct 18, 2021 at 14:20 — user15980977
– user15980977, Commented Oct 18, 2021 at 14:20

cruisepandey · Accepted Answer · 2021-10-18 14:23:39Z

2

There are many ways to handle stale element reference.

One is like try to re-click on the web element in a while loop.

Your link_text also looks wrong, Please use the below xpath :

# click cookies popup
driver.find_element_by_xpath('//*[(@id = "description")]//*[contains(concat( " ", @class, " " ), concat( " ", "tc-open-privacy-center", " " ))]').click()

time.sleep(10)

# click show more button until no more results to load
while True:
    try:
        more_button = wait.until(EC.visibility_of_element_located((By.XPATH, "//a[starts-with(@onclick,'tagDeClick') and contains(@href,'/offres/emploi.rechercheoffre:afficherplusderesultats')]")))
        ActionChains(driver).move_to_element(more_button).perform()
        attempts = 0
        while attempts < 2 :
            try:
                more_button.click()
                break
            except StaleElementReferenceException as exception:
                print(exception.msg)
            attempts = attempts  + 1

    except TimeoutException:
        break

time.sleep(10)

print(driver.page_source)
print("Complete")

time.sleep(10)

Output :

stale element reference: element is not attached to the page document
  (Session info: chrome=94.0.4606.81)

If you see this in logs, and you do not wish to see this, you will have to comment print(exception.msg).

Imports :

from selenium.webdriver.common.action_chains import ActionChains

answered Oct 18, 2021 at 14:23

cruisepandey

29.5k6 gold badges23 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

hug Over a year ago

Thanks. Do you have an idea why I have a undefined name 'StaleElementReferenceException' at the line except StaleElementReferenceException as exception ?

cruisepandey Over a year ago

you have to import from selenium.common.exceptions import StaleElementReferenceException this should help you past the undefined exception.

Nandan A · Accepted Answer · 2021-10-18 14:10:19Z

2

StaleElementReferenceException: stale element reference: element is not attached to the page document

Indicates that a reference to an element is now "stale" --- the element no longer appears on the DOM of the page. The reason for this expectation is may be your DOM got updated or refreshed. For an example, after performing an action like click() your DOM may get updated or refreshed. In this time when you are trying to find an element on DOM you will experience this error.

You have to re-find that element in updated or refreshed DOM

       try:  
            more_button = wait.until(EC.visibility_of_element_located((By.LINK_TEXT, 'AFFICHER LES 20 OFFRES SUIVANTES'))).click()  
     except StaleElementReferenceException:
            more_button = WebDriverWait(self.driver, 10).until(EC.visibility_of_element_located((By.LINK_TEXT, 'AFFICHER LES 20 OFFRES SUIVANTES')))
            more_button.click()

answered Oct 18, 2021 at 14:10

Nandan A

2,9721 gold badge16 silver badges27 bronze badges

1 Comment

Nandan A Over a year ago

You haven't mentioned at what line the exception occurs. Please see and add this exception handling at that step.

score 0 · Accepted Answer · 2021-10-18 14:39:12Z

Try using the execute_script method i think it is the most reliable method to solve this kind of problem.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException, NoSuchElementException
import time

options = webdriver.ChromeOptions()
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')

site = 'https://candidat.pole-emploi.fr/offres/emploi/horticulteur/s1m1'
wd = webdriver.Chrome("C:\Program Files (x86)\chromedriver.exe", options=options)
wd.get(site)

time.sleep(10)

wait = WebDriverWait(wd, 10)

# click cookies popup
wd.find_element_by_xpath('//*[(@id = "description")]//*[contains(concat( " ", @class, " " ), concat( " ", "tc-open-privacy-center", " " ))]').click()

time.sleep(10)

# click show more button until no more results to load
while True:
    try:
        wait.until(EC.visibility_of_element_located((By.LINK_TEXT, 'AFFICHER LES 20 OFFRES SUIVANTES')))
        more_button = wd.find_element_by_link_text('AFFICHER LES 20 OFFRES SUIVANTES')
        wd.execute_script('arguments[0].click()', more_button)
        #print('clicked')
    except (TimeoutException, NoSuchElementException):
        break

time.sleep(10)

print(wd.page_source)
print("Complete")

time.sleep(10)
wd.quit()

Collectives™ on Stack Overflow

StaleElementReferenceException issue while scraping with Selenium

3 Answers 3

2 Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related