find element by xpath not defined

Question

When I run

# change the value inside the range to save the number of reviews we're going to grab
for i in range(0, pages_to_scrape):

    # give the DOM time to load
    time.sleep(2) 

    # Click the "expand review" link to reveal the entire review.
    find_element_by_xpath(".//div[contains(@data-test-target, 'expand-review')]").click()

    # Now we'll ask Selenium to look for elements in the page and save them to a variable. First lets define a container that will hold all the reviews on the page. In a moment we'll parse these and save them:
    container = driver.findElement(By.xpath("//div[@data-reviewid]"))

I get the error

NameError                                 Traceback (most recent call last)
/var/folders/6c/jpl964752rv_72zjclrp_8ym0000gn/T/ipykernel_18848/426937913.py in <module>
      6 
      7     # Click the "expand review" link to reveal the entire review.
----> 8     find_element_by_xpath(".//div[contains(@data-test-target, 'expand-review')]").click()
      9 
     10     # Now we'll ask Selenium to look for elements in the page and save them to a variable. First lets define a container that will hold all the reviews on the page. In a moment we'll parse these and save them:

NameError: name 'find_element_by_xpath' is not defined

I used print(dir(driver)) to see what command i'm supposed to use, and the one I am using is the correct one. I was originally using driver.findElement(By.xpath(".//div[contains(@data-test-target, 'expand-review')]").click(), which worked, but when combined with other codes gave the same error above, except it said "By" was not defined. Then I tried a user suggestion driver.find_element(By.XPATH, 'your xpath'), which also generated the same error.

Full code:

import selenium
import csv #This package lets us save data to a csv file
from selenium import webdriver #The Selenium package we'll need
import time #This package lets us pause execution for a bit

path_to_file = "/Users/user/Desktop/HotelReviews.csv"

pages_to_scrape = 3

url = "https://www.tripadvisor.com/Hotel_Review-g60982-d209422-Reviews-Hilton_Waikiki_Beach-Honolulu_Oahu_Hawaii.html"

# import the webdriver
driver = webdriver.Safari()
driver.get(url)

# open the file to save the review
csvFile = open(path_to_file, 'a', encoding="utf-8")
csvWriter = csv.writer(csvFile)

# change the value inside the range to save the number of reviews we're going to grab
from selenium.webdriver.common.by import By
for i in range(0, pages_to_scrape):

    # give the DOM time to load
    time.sleep(2) 

    # Click the "expand review" link to reveal the entire review.
    driver.find_elements(by=By.XPATH, ".//div[contains(@data-test-target, 'expand-review')]").click()

  
# When all pages have been processed, quit the driver
driver.quit()

EDIT: Here is the updated code with new errors

from selenium.webdriver.common.by import By
for i in range(0, pages_to_scrape):

# give the DOM time to load
time.sleep(2) 

# Click the "expand review" link to reveal the entire review.
driver = webdriver.Safari() #url was defined in a different portion of the code
driver.get(url)
driver.find_element_by_xpath(".//div[contains(@data-test-target, 'expand-review')]").click()

which generates

  File "/var/folders/6c/jpl964752rv_72zjclrp_8ym0000gn/T/ipykernel_24069/1057492970.py", line 15
    driver.find_elements(by=By.XPATH,".//div[contains(@data-test-target, 'expand-review')]").click()
                                                                                           ^
SyntaxError: positional argument follows keyword argument

Following the format in the error code, I ran

driver.find_elements(by=By.XPATH, ".//div[contains(@data-test-target, 'expand-review')]").click()

Which then gave another error:

File "/var/folders/6c/jpl964752rv_72zjclrp_8ym0000gn/T/ipykernel_24069/4123117837.py", line 11
    driver.find_elements(by=By.XPATH, ".//div[contains(@data-test-target, 'expand-review')]").click()
                                                                                            ^
SyntaxError: positional argument follows keyword argument

You are mixing some Java in there as well.

Arundeep Chohan
– Arundeep Chohan

2022-03-13 11:22:54 +00:00
Commented Mar 13, 2022 at 11:22 — Arundeep Chohan
– Arundeep Chohan, Commented Mar 13, 2022 at 11:22

JimShapedCoding · Accepted Answer · 2022-03-13 08:22:25Z

1

You are calling find_element_by_xpath directly.

All the selenium methods need to be called from the driver object after you instantiate it, declaring the website you are scraping.

So the steps are:

Create the driver instance
Use get() to go to the website you want to scrape
Find your elements by XPath. But, refer to the methods from the driver instance

Example:

driver = webdriver.Chrome("path/to/driver")
driver.get("yourwebsite.com")
driver.find_element_by_xpath(".//div[contains(@data-test-target, 'expand-review')]").click()

answered Mar 13, 2022 at 8:22

JimShapedCoding

9371 gold badge7 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

bandcar Over a year ago

Hello everyone! I updated the code, but it generated new errors. I added it to my original post

bandcar Over a year ago

Also added the full code I'm using!

bandcar Over a year ago

My post got extremely long. I created a new one with updated information, so to avoid confusion, the new post is here

si ho · Accepted Answer · 2024-03-27 06:51:37Z

0

Use this line instead:

from selenium.webdriver.common.by import By

answered Mar 27, 2024 at 6:51

si ho

1

Collectives™ on Stack Overflow

find element by xpath not defined

EDIT: Here is the updated code with new errors

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

EDIT: Here is the updated code with new errors

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related