Error while using XPath in Selenium, python

Question

I am trying to access the name and the link of restaurants on a website. Although i copied the xpath from the page source, i still get the error:

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="component_2"]/div/div[2]/span/div[2]/div[1]/div/div/a"} (Session info: chrome=78.0.3904.87)

browser.find_element_by_class_name('submit_text').click()
browser.implicitly_wait(5)
paths = []
for i in range(2,14):
    if i%6 != 0:
        paths.append(browser.find_element_by_xpath('//*[@id="component_2"]/div/div['+str(i)+']/span/div[2]/div[1]/div/div/a'))
link = []
restaurant_name = []
for i in range(10):
    element = browser.find_element_by_xpath(paths[i])
    restaurant_name.append(element.get_attribute('innerHTML'))
    link.append(element.get_attribute('href')) 


print(paths)

Here is the HTML for the element I am trying to locate:

<a href="/Restaurant_Review-g1080422-d10195584-Reviews-Restaurant_Brau-Sant_Cugat_del_Valles_Catalonia.html" class="restaurants-list-ListCell__restaurantName--2aSdo" target="_self">
    1. Restaurant Brau
</a>

It sounds like your locator strategy needs to be changed here -- your XPath is not locating an element. The XPath from the page source can be very brittle and does not take dynamic page content into account. We probably need to see some HTML from the page or link to the page you are automating so that we can help you fix your locator. — CEH
– CEH, Commented Nov 11, 2019 at 19:53
<a href="/Restaurant_Review-g1080422-d10195584-Reviews-Restaurant_Brau-Sant_Cugat_del_Valles_Catalonia.html" class="restaurants-list-ListCell__restaurantName--2aSdo" target="_self">1. Restaurant Brau</a> — programmerA
– programmerA, Commented Nov 11, 2019 at 19:58
The above is from the page source, and is the part that i am essentially trying to reach, however I have to get 10 of these elements, that is why i tried to use the for loop — programmerA
– programmerA, Commented Nov 11, 2019 at 19:59
This is the link: tripadvisor.com/… I need the name and link of the top10 restaurants — programmerA
– programmerA, Commented Nov 11, 2019 at 20:00
Thanks for the extra detail, I've edited your post to include that in your question so everyone can see it. The issue might be with the quotation mark strategy used in your XPath -- it seems like single and double quotes are getting mixed in where they shouldn't be. I posted a simple solution fixing this -- in the meantime, I will check out the page source from the website you provided and see what I can test out there. — CEH
– CEH, Commented Nov 11, 2019 at 20:01

KunduK · Accepted Answer · 2019-11-11 20:30:15Z

Induce WebDriverWait and visibility_of_all_elements_located() and use following css selector.

This will retrieve all 31 restaurant on page 1

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver=webdriver.Chrome()
driver.get("https://www.tripadvisor.co.uk/Restaurants-g1080422-Sant_Cugat_del_Valles_Catalonia.html")
restaurants=WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,".restaurants-list-List__wrapper--3PzDL a.restaurants-list-ListCell__restaurantName--2aSdo")))
for restaurant in restaurants:
    print("Restaurant Name : " + restaurant.text)
    print("Restaurant URL : " + restaurant.get_attribute("href"))

Output:

Restaurant Name : VdGUST
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d12687182-Reviews-VdGUST-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 1. Restaurant Brau
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d10195584-Reviews-Restaurant_Brau-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 2. La Rita
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d10365477-Reviews-La_Rita-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 3. Kitsune Sushi Bar
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d5966644-Reviews-Kitsune_Sushi_Bar-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 4. Sabatic
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d10167691-Reviews-Sabatic-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 5. Dakidaya
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d4546707-Reviews-Dakidaya-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 6. El Vi de Deu
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d7076969-Reviews-El_Vi_de_Deu-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 7. Nemesis Gastronomia
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d11892809-Reviews-Nemesis_Gastronomia-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 8. La Rampa 23
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d4043880-Reviews-La_Rampa_23-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 9. Bodega Tomas
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d7912636-Reviews-Bodega_Tomas-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 10. Dos Cucharas Restaurant
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d17406200-Reviews-Dos_Cucharas_Restaurant-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 11. Restaurante 9 Reinas
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d7155184-Reviews-Restaurante_9_Reinas-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 12. Sugoi
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d2241212-Reviews-Sugoi-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 13. Il Fornetto
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d3571022-Reviews-Il_Fornetto-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 14. La Tartine
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d9702827-Reviews-La_Tartine-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 15. La Pina de Plata
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d3981558-Reviews-La_Pina_de_Plata-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 16. La Burgueseria
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d3452640-Reviews-La_Burgueseria-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 17. Vulcano Restaurant
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d2389165-Reviews-Vulcano_Restaurant-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 18. Pizzeria La Bota
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d5771344-Reviews-Pizzeria_La_Bota-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 19. Braseria La Bolera
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d2276369-Reviews-Braseria_La_Bolera-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 20. El Meson
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d6978197-Reviews-El_Meson-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 21. Le Ciel Sant Cugat
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d15129074-Reviews-Le_Ciel_Sant_Cugat-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 22. Foodies
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d6785484-Reviews-Foodies-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 23. Margotin
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d2482063-Reviews-Margotin-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 24. Piaceri D’ Italia Ristorante Pizzeria
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d19084243-Reviews-Piaceri_D_Italia_Ristorante_Pizzeria-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 25. L'antic de la placa
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d6673634-Reviews-L_antic_de_la_placa-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 26. Les Voltes
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d11897574-Reviews-Les_Voltes-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 27. El Mirador de Can Cases
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d2258911-Reviews-El_Mirador_de_Can_Cases-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 28. Casa Blava
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d3351146-Reviews-Casa_Blava-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 29. LABARRA San Cugat
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d4103134-Reviews-LABARRA_San_Cugat-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Name : 30. Pizzeria Parandu
Restaurant URL : https://www.tripadvisor.co.uk/Restaurant_Review-g1080422-d8291736-Reviews-Pizzeria_Parandu-Sant_Cugat_del_Valles_Catalonia.html

CEH · Accepted Answer · 2019-11-11 20:13:05Z

0

After running your XPath query against the page link you provided, no results were retrieved. You mentioned you need to get the name and link of top 10 restaurants.

You are on the right track with a loop, but I would modify it a bit to iterate a list of WebElement instead of generating locators within the loop:

# find list of all restaurant links on page
# this retrieves 24 restaurant links
restaurant_link_list = browser.find_elements_by_xpath("//div[contains(@class, 'restaurant_shelf_item')]/div/a")

# find list of all restaurant names on page
# this retrieves 24 restaurant names
restaurant_name_list = browser.find_elements_by_xpath("//div[contains(@class, 'restaurant_shelf_item')]/div/div/div[@class='item name']")

# loop through first 10 restaurants and print their name / link
for i in range(0, 10):

    # get restaurant link
    link = restaurant_link_list[i].get_attribute("href")
    print(link)

    # get restaurant name
    name = restaurant_name_list[i].get_attribute("title")
    print(name)

The output is:

https://www.tripadvisor.com/Restaurant_Review-g1080422-d10167691-Reviews-Sabatic-Sant_Cugat_del_Valles_Catalonia.html
Sabatic
https://www.tripadvisor.com/Restaurant_Review-g1080422-d7076969-Reviews-El_Vi_de_Deu-Sant_Cugat_del_Valles_Catalonia.html
El Vi de Deu
https://www.tripadvisor.com/Restaurant_Review-g1080422-d4546707-Reviews-Dakidaya-Sant_Cugat_del_Valles_Catalonia.html
Dakidaya
https://www.tripadvisor.com/Restaurant_Review-g1080422-d11892809-Reviews-Nemesis_Gastronomia-Sant_Cugat_del_Valles_Catalonia.html
Nemesis Gastronomia
https://www.tripadvisor.com/Restaurant_Review-g1080422-d3981558-Reviews-La_Pina_de_Plata-Sant_Cugat_del_Valles_Catalonia.html
La Pina de Plata
https://www.tripadvisor.com/Restaurant_Review-g1080422-d10694292-Reviews-Masia_Can_MagI-Sant_Cugat_del_Valles_Catalonia.html
Masia Can MagI
https://www.tripadvisor.com/Restaurant_Review-g1080422-d2281259-Reviews-Bocca_Restaurant_Club-Sant_Cugat_del_Valles_Catalonia.html
Bocca Restaurant & Club
https://www.tripadvisor.com/Restaurant_Review-g1080422-d10733699-Reviews-Andonie_pastissers-Sant_Cugat_del_Valles_Catalonia.html
Andonie pastissers
https://www.tripadvisor.com/Restaurant_Review-g1080422-d10195584-Reviews-Restaurant_Brau-Sant_Cugat_del_Valles_Catalonia.html
Restaurant Brau
https://www.tripadvisor.com/Restaurant_Review-g1080422-d10365477-Reviews-La_Rita-Sant_Cugat_del_Valles_Catalonia.html
La Rita

Note that for reading static content from a page as such, you may find it more efficient to use the Python Requests library -- however, if you wish to using Selenium, this code will accomplish what you are looking to do.

edited Nov 11, 2019 at 20:13

answered Nov 11, 2019 at 19:56

CEH

5,9142 gold badges19 silver badges43 bronze badges

5 Comments

programmerA Over a year ago

Thank you for the advice. I will keep it in mind for future use. Unfortunately this way doesn't work either, i get the same error message

CEH Over a year ago

Updated my answer with a different solution which changes your selector and looping strategy.

programmerA Over a year ago

Thank you for the response. The answer may not be the exact items I was looking for but your strategy is very helpful indeed! I will try to adapt it to my needs. Thank you very much!

programmerA Over a year ago

I will explore the option with Python Request as well, thanks

CEH Over a year ago

If this solution ends up helping you solve your problem, feel free to mark as the answer to your question so other users can see that it was helpful.

Collectives™ on Stack Overflow

Error while using XPath in Selenium, python

2 Answers 2

1 Comment

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related