Selenium can't detect text in html from Google Maps

Question

I am trying to scrape travel times between two points from Google Maps:

I have used inspect to find the XPath of the travel time in the HTML file:

The following code raises an exception:

from selenium.webdriver.chrome.service import Service
from selenium import webdriver
from selenium.webdriver.common.by import By

url = 'https://www.google.com/maps/dir/35.7290613,+51.238014/35.7504171,51.2244444/@35.7296931,51.2580754,13z/data=!4m8!4m7!1m5!1m1!1s0x0:0x3a1893ebcae30b2e!2m2!1d51.238014!2d35.7290613!1m0'

#initialize web driver
with webdriver.Chrome() as driver:
    #navigate to the url
    driver.get(url)
    #find element by xpath
    myDiv = driver.find_element(By.XPATH, '/html/body/div/div[12]/div[2]/div/div[2]/div/div[3]/button[1]/div/div[1]/div/h1/span[2]/span/span')
    print(myDiv)
    print(myDiv.text)

The exception:

NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div/div[12]/div[2]/div/div[2]/div/div[3]/button[1]/div/div[1]/div/h1/span[2]/span/span"} (Session info: chrome=108.0.5359.125)

Are there any workarounds?

Prophet · Accepted Answer · 2022-12-20 15:05:40Z

1

First, you need to wait for the page to becopme loaded.
Then you need to located the correct elements. The elements you want to scrape can be located by this XPath:

"//div[contains(@aria-labelledby,'section-directions-trip-title')]//span[@jstcache='198']"

I see 3 traveling options on that page, all the 3 trip times can be received as folowing:

trip_times = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(@aria-labelledby,'section-directions-trip-title')]//span[@jstcache='198']")))
for trip_time in trip_times:
    print(trip_time.text)

The complete code is as following and it worked correct:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

options = Options()
options.add_argument("start-maximized")

webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 30)

url = "https://www.google.com/maps/dir/35.7290613,+51.238014/35.7504171,51.2244444/@35.7296931,51.2580754,13z/data=!4m8!4m7!1m5!1m1!1s0x0:0x3a1893ebcae30b2e!2m2!1d51.238014!2d35.7290613!1m0"
driver.get(url)

trip_times = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(@aria-labelledby,'section-directions-trip-title')]//span[@jstcache='198']")))
for trip_time in trip_times:
    print(trip_time.text)

edited Dec 20, 2022 at 15:05

answered Dec 20, 2022 at 14:16

Prophet

33.5k28 gold badges58 silver badges90 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

ali bakhtiari Over a year ago

Thank you for your answer. How can I detect the element with the correct XPath form? Any references?

ali bakhtiari Over a year ago

The code sometimes raises the following exception:

Traceback (most recent call last):    File "<ipython-input-12-545316d94ef4>", line 18, in <module>     trip_times = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//div[contains(@aria-labelledby,'section-directions-trip-title')]//span[@jstcache='198']")))    File "C:\Users\ASUS\anaconda3\lib\site-packages\selenium\webdriver\support\wait.py", line 95, in until     raise TimeoutException(message, screen, stacktrace)  TimeoutException

does that have to do with my internet connection?

Prophet Over a year ago

Sorry, I'm not sure I correctly understand your first question. As about the timeout - my fault. We normally use 20 or 30 or even 60 seconds timeout. This normally will not make problems since once element / expected condition is found the code will continue instantly while 20 - 30 seconds are normally enough to pages to be loaded. SO, for slow internet timeout of 5 seconds may indeed be too short.

Collectives™ on Stack Overflow

Selenium can't detect text in html from Google Maps

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related