How to scrape information inside an unordered list selenium + python

Question

I am working on a web scraping project, where I try to scrape information from the amazon website. In the website, there exists an unordered list with such information

Item Weight: 17.2 pounds
Shipping Weight: 17.4 pounds (View shipping rates and policies)
ASIN: B00HC767P6
UPC: 766789717088 885720483186 052000201628
Item model number: mark-1hooi-toop842
Customer Reviews: 4.8 out of 5 stars1,352 customer ratings
Amazon Best Sellers Rank: #514 in Grocery & Gourmet Food (See Top 100 in Grocery & Gourmet Food)
#12 in Sports Drinks

The list itself does not have any class to it. The problem is I do not want all the information from the list. Only the ASIN code. the li tags do not have any specific class or ID to them. here is the link to the product details page

Before selenium, I was working with BeautifulSoup and this is how I tackled the issue

asin = str(soup.find('bdi', {'dir': 'ltr'}).find_parent('li'))[38:].split('<')[0]

I am now switching to selenium. How do I scrape the information.

Mesut GUNES · Accepted Answer · 2020-06-05 10:21:57Z

1

You can use the css selector to get the related li item as follow:

Finding the child element by index by css selector

$(".content > ul > li:nth-child(2)").textContent >>> "Shipping Weight: 33 pounds (View shipping rates and policies)"
$(".content > ul > li:nth-child(3)").textContent >>> "ASIN: B07QKN2ZT9"

related python selenium code:

driver.find_element_by_css_selector(".content > ul > li:nth-child(3)").text.split(": ")[1] >>> 'B07QKN2ZT9'

Finding the ancestors element by XPATH

If the ASIN is not always in the same index, then you can find the bdi element that has text ASIN text and find its ancestor::li then get its text and extract the related part. Like the following:

driver.find_element_by_xpath("//bdi[text()='ASIN']/ancestor::li").text.split(": ")[1] >>> 'B07QKN2ZT9'

Generate XPATH

//<element type>[<attribute type> = <attribute value>]/<descendant>
//bdi[text() = 'ASIN'] >>> bdi element with text 'ASIN'
//bdi[@dir = 'ltr'] >>> bdi element with dir attribute equals to 'ltr'

Access to an ancestor of an element

/ancestor::<ancestor element type>
//bdi[text()='ASIN']/ancestor::li >>> li
//bdi[text()='ASIN']/ancestor::ul >>> ul

You can check this as a reference

edited Jun 5, 2020 at 10:21

answered Jun 2, 2020 at 11:12

Mesut GUNES

7,4612 gold badges37 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Sashaank Over a year ago

Hi. While this answer works, the asin field might not always be the 3rd element in the list. That is the problem with this. sometimes it might also be the 2nd element and sometimes the 4th. I will need to access only that field in all cases...

Mesut GUNES Over a year ago

@Sashaank this was not mentioned but updated the solution you can check it

Sashaank Over a year ago

Thanks a lot. this code works like a charm. I will have to start learning xpath more properly now. If it is possible can you point me to some good tutorial on xpath?

Mesut GUNES Over a year ago

added a reference

mnm Over a year ago

@MesutGUNES what is >>> ? in fact, pls can you explain this entire line of code driver.find_element_by_xpath("//bdi[text()='ASIN']/ancestor::li").text.split(": ")[1] >>> 'B07QKN2ZT9'in terms of what is it doing?

|

Collectives™ on Stack Overflow

How to scrape information inside an unordered list selenium + python

1 Answer 1

Finding the child element by index by css selector

Finding the ancestors element by XPATH

Generate XPATH

Access to an ancestor of an element

7 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Finding the child element by index by css selector

Finding the ancestors element by XPATH

Generate XPATH

Access to an ancestor of an element

7 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related