How to scrape li tag in selenium python with class name spaces involved?

Question

<div class="all__BrandsAllListGrid-sc-1410m2d-5 eWrRoU"><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/00-seeds">00 Seeds</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/1-luv-farms">1 Luv Farms</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/1-lyfe">1 Lyfe</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/1-puff">1 PUFF</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/101-cannabis">101 Cannabis</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/101-cbd">101 CBD</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/1111cannabiscompany">11:11 Cannabis Company</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/1849-extracts">1849 Extracts</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/18twelve">18twelve</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/1906-new-highs">1906 New Highs</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/1937-vireo">1937 Cannabis</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/1937-concentrates">1937 Concentrates</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/19forty">19Forty Los Angeles</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/1cbd">1CBD</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/1lyfe">1Lyfe</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/22-red">22 Red</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/220-farms">220 Farms</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/27-life">27 Life</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/27-pure">27 Pure</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/2wisted">2wisted</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/3-bros">3 Bros Grow</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/3-docs-extracts">3 DOCS EXTRACTS</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/3-leaf">3 Leaf</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/brand-3-2-4">324</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/34-street-seed-co">34 Street Seed Co.</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/3c-farms">3C Farms</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/3d-farms">3D Farms</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/3leaf-edibles">3Leaf Edibles</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/420-bomb">420 Bomb</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/420-green-cannabis">420 Green Cannabis</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/420me">420ME</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/48north">48North</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/4score">4SCORE</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/5-leaf">5 Leaf</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/528-cannabis-co">528 Cannabis Co.</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/7-points-oregon">7 Points Oregon</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/710-labs">710 Labs</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/7acres">7ACRES</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/7leafok">7LeafOK</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/818-brands">818 Brands</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/818-genetics">818 Genetics</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/9-trees">9Trees</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/a-golden-state">A Golden State</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/emg-distribution">A to B Gardens</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/a-extract">A+ Extract</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/a-liv">A-LIV</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/a88cbd">A88CBD</a></li><li class="all__BrandsAllListItem-sc-1410m2d-6 kLYBAG"><a href="/brands/abba-medix">Abba Medix</a></li></div>

I have implemented this way but didn't work out.

list = driver.find_elements_by_xpath('//*[@id="content"]/div[2]/div[2]/ul/div/li[1]/a')
list = [li.get_attribute('href') for li in list]
print(list)

How to scrape all li tags?

Prophet · Accepted Answer · 2021-06-03 13:27:21Z

1

Try this:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

#wait until first element is visible
wait.until(EC.visibility_of_element_located((By.XPATH, '//li[contains(@class,'all__BrandsAllListItem')]')))
#add some more delay to make sure all the other elements are loaded
time.sleep(1)

list = driver.find_elements_by_xpath('//li[contains(@class,'all__BrandsAllListItem')]/a')
list = [a.get_attribute('href') for a in list]
print(list)

edited Jun 3, 2021 at 13:27

answered Jun 3, 2021 at 12:46

Prophet

33.5k29 gold badges58 silver badges90 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Gaurav Over a year ago

It throws [] empty list. @prophet

Prophet Over a year ago

Probably you are missing some delay / wait before that to let the elements loaded

Gaurav Over a year ago

Yes. It is the issue. How can I resolve that delay or missing wait?

Prophet Over a year ago

I have added some example. The best practice is to use explicit wait. However since there is no way to know when ALL the elements are visible I added some additional delay to make all the elements loaded before getting the elements list

Prophet Over a year ago

You are welcome! But you do not need to put a fixed sleep of 20 seconds. wait.until(EC.visibility_of_element_located will wait exactly the time the page is loaded. I added 1 second fixed delay only to make sure all the other elements are loaded too

|

cruisepandey · Accepted Answer · 2021-06-03 13:21:29Z

0

try this :

driver.get("https://weedmaps.com/brands/all")
all_li = driver.find_elements(By.XPATH, "//li")
for li in all_li:
    print(li.get_attribute('innerHTML'))

If you just want href use below code :

driver.get("https://weedmaps.com/brands/all")
all_li = driver.find_elements(By.XPATH, "//li/a")
for li in all_li:
    print(li.get_attribute('href'))

answered Jun 3, 2021 at 13:21

cruisepandey

29.5k6 gold badges23 silver badges43 bronze badges

2 Comments

Gaurav Over a year ago

I have tried this but it throws only the first li tags only like menu only.

cruisepandey Over a year ago

What is the expected outcome ? The URL that you have shared only contains 8 li tag in that particular page, Do you want all the href from a tag irrespective of li tag ?

Collectives™ on Stack Overflow

How to scrape li tag in selenium python with class name spaces involved?

2 Answers 2

6 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related