Scraping loop on list with next page using Selenium

Question

is somewhat similar althogh i dont want to make it more complicated by using with bs4

I want to scrape all 10 titles (links to pdfs) on every page. Here is the code that throws a StaleElementReferenceException: Message: stale element reference: element is not attached to the page document (at i.click())

driver.find_element_by_css_selector('button[class="btn btn-primary"][type="submit"]').click()    
    
for j in range(10):
    pdfs = driver.find_elements_by_css_selector(".title a")
    if j < 10:
        nextpg = driver.find_element_by_css_selector(".tx-pagebrowse-page button")
        pdfs = driver.find_elements_by_css_selector(".title a")
    for i in pdfs:
        i.click()
        time.sleep(5)
    if j < 10:
        nextpg.click()
    else:
        break

below it shows that the first page is a type=list list with the right titles

pdfs = driver.find_elements_by_css_selector(".title a")
print(type(pdfs))
for my_pdf in pdfs:
    print(my_pdf.get_attribute('innerHTML'))

<class 'list'>
                                dfhgadr asfdgsd
                                sDF SDF 
                                SDFSDF sdf
                                SDF DSfdf
                                SDFD sdf
                                SDFSdf fdf
                                zewr4
                                sHSDFG 
                                RSZJRT
                                dhff fgf

Md. Fazlul Hoque · Accepted Answer · 2021-07-10 16:15:43Z

0

Instead of pdfs[[i]].click() You may try as follows:

 i.click()

answered Jul 10, 2021 at 16:15

Md. Fazlul Hoque

16.2k5 gold badges15 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

id345678 Over a year ago

Hi. ´AttributeError: 'int' object has no attribute 'click'´

Md. Fazlul Hoque Over a year ago

Try to use as: str(i).click()

id345678 Over a year ago

AttributeError: 'str' object has no attribute 'click'

Md. Fazlul Hoque Over a year ago

Instead of clic() , try to use Send_keys(Keys.ENTER)

Md. Fazlul Hoque Over a year ago

Because element may remove from DOM.Please take help fron here:stackoverflow.com/questions/18225997/…

|

Collectives™ on Stack Overflow

Scraping loop on list with next page using Selenium

1 Answer 1

10 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

10 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related