Web scraping ignore "Next" or ">" when hidden (Selenium, Python)

Question

I am using Selenium for Python to scrape a site with multiple pages. To get to the next page, I use driver.find_element(By.XPATH, xpath). However, The xpath text changes. So, instead, I want to use other attributes.

I tried to find by class, using "page-link": driver.find_element(By.CLASS_NAME, "page-link". However, the "page-link" class is also present in the disabled list item. As a result, the Selenium driver won't stop after the last page, in this case page 2.

I want to stop the driver clicking the disabled item on the page, i.e. I want it to ignore the last item in the list, the one with "page-item disabled", aria-disabled="true" and aria-hidden="true". The idea is that if the script can't find that item, it will end a while loop that relies on the ">" button to be enabled.

See the source code below.

Please advise.

<nav>
<ul class="pagination">
<li class="page-item">
    <a class="page-link" href="https://www.blucap.net/app/FlightsReport?fromdate=2023-02-01&amp;todate=2023-02-28&amp;filterByMemberId=&amp;view=View%20Report&amp;page=1" rel="prev" aria-label="&laquo; Previous">&lsaquo;</a>
</li>
<li class="page-item">
    <a class="page-link" href="https://www.blucap.net/app/FlightsReport?fromdate=2023-02-01&amp;todate=2023-02-28&amp;filterByMemberId=&amp;view=View%20Report&amp;page=1">1</a>
</li>
<li class="page-item active" aria-current="page">
    <span class="page-link">2</span>
</li>
<li class="page-item disabled" aria-disabled="true" aria-label="Next &raquo;">
    <span class="page-link" aria-hidden="true">&rsaquo;</span>
</li>
</ul>
</nav>

undetected Selenium · Accepted Answer · 2023-02-19 20:34:46Z

0

To go to the Next Page there can be a couple of approaches:

You can opt to find_element() and click it's descendant <span> of the <li> with aria-label="Next »" but doesn't contains aria-disabled="true" as follows:
```
driver.find_element(By.XPATH, "//li[starts-with(@aria-label, 'Next') and not(@aria-disabled='true')]/span").click()
```

answered Feb 19, 2023 at 20:34

undetected Selenium

194k44 gold badges304 silver badges387 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Martien Lubberink · Accepted Answer · 2023-02-20 18:56:22Z

0

Alas, the solution offered did not work.

In the end I decided to rely on the http links and, using regex, extract the pages with page larger than 1 (as the first page is the starting page). Works like a charm.

Thanks for any effort, much appreciated.

answered Feb 20, 2023 at 18:56

Martien Lubberink

2,7671 gold badge23 silver badges35 bronze badges

Collectives™ on Stack Overflow

Web scraping ignore "Next" or ">" when hidden (Selenium, Python)

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related