Selenium find all elements by xpath

Question

I used selenium to scrap a scrolling website and conducted the code below

import requests
from bs4 import BeautifulSoup
import csv
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
import unittest
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import time
import unittest
import re

output_file = open("Kijubi.csv", "w", newline='')  

class Crawling(unittest.TestCase):
    def setUp(self):
        self.driver = webdriver.Firefox()
        self.driver.set_window_size(1024, 768)
        self.base_url = "http://www.viatorcom.de/"
        self.accept_next_alert = True

    def test_sel(self):
        driver = self.driver
        delay = 3
        driver.get(self.base_url + "de/7132/Seoul/d973-allthingstodo")
        for i in range(1,1):
            driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
            time.sleep(2)
    html_source = driver.page_source
    data = html_source.encode("utf-8")

My next step was to crawl specific information from the website like the price.

Hence, I added the following code:

 all_spans = driver.find_elements_by_xpath("/html/body/div[5]/div/div[3]/div[2]/div[2]/div[1]/div[1]/div")
    print(all_spans)
    for price in all_spans:
        Header = driver.find_elements_by_xpath("/html/body/div[5]/div/div[3]/div[2]/div[2]/div[1]/div[1]/div/div[2]/div[2]/span[2]")
        for span in Header:
            print(span.text)

But I get just one price instead all of them. Could you provide me feedback on what I could improve my code? Thanks:)

EDIT

Thanks to your guys I managed to get it running. Here is the additional code:

    elements = driver.find_elements_by_xpath("//div[@id='productList']/div/div")

    innerElements = 15

    outerElements = len(elements)/innerElements

    print(innerElements,  "\t", outerElements, "\t", len(elements))

    for j in range(1, int(outerElements)):

        for i in range(1, int(innerElements)):


            headline = driver.find_element_by_xpath("//div[@id='productList']/div["+str(j)+"]/div["+str(i)+"]/div/div[2]/h2/a").text

            price = driver.find_element_by_xpath("//div[@id='productList']/div["+str(j)+"]/div["+str(i)+"]/div/div[2]/div[2]/span[2]").text
            deeplink = driver.find_element_by_xpath("//div[@id='productList']/div["+str(j)+"]/div["+str(i)+"]/div/div[2]/h2/a").get_attribute("href")

            print("Header: " + headline + " | " + "Price: " + price + " | " + "Deeplink: " + deeplink)

Now my last issue is that I still do not get the last 20 prices back, which have a English description. I only get back the prices which have German description. For English ones, they do not get fetched although they share the same html structure.

E.g. html structure for the English items

     headline =   driver.find_element_by_xpath("//div[@id='productList']/div[6]/div[1]/div/div[2]/h2/a")

Do you guys know what I have to modify? Any feedback is appreciated:)

The XPath you have is only targeting one element - that XPath has been copied/auto-generated, so I'd suggest taking a look at the XPath spec & tutorials so you can understand it. — Arran
– Arran, Commented Aug 17, 2015 at 15:39

Community · Accepted Answer · 2017-05-23 11:46:42Z

8

To grab all prices on that page you should use such XPATH:

Header = driver.find_elements_by_xpath("//span[contains(concat(' ', normalize-space(@class), ' '), 'price-amount')]")

which means: find all span elements with class=price-amount, why so complex - see here

But more simply to find the same elements is by CSS locator:

.price-amount

edited May 23, 2017 at 11:46

CommunityBot

11 silver badge

answered Aug 18, 2015 at 16:36

Viktor Chmel

1447 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Serious Ruffy Over a year ago

Thanks for your feedback

Innovations Anonymous Over a year ago

WebDriver object has no attribute find_elements_by_xpath

Viktor Chmel Over a year ago

In Python it has - geeksforgeeks.org/…

Collectives™ on Stack Overflow

Selenium find all elements by xpath

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related