Google search with Selenium Python doesn't work

Question

I would like to retrieve the phone number of some points of interest from the Google results page with Selenium Python. I can accept the terms of use of Google but I can neither enter the query nor press the button. This is my code:

rom selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

url = 'https://www.google.it/'

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 20)
driver.get(url)

time.sleep(0.5)

# Accept the Google terms of use
driver.execute_script('return document.querySelector("#L2AGLb > div")').click()

names = ['Ristorante Roma Antica Roma', 'Ristorante e Braceria Al Piave Roma']

for name in names:
    input_search = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'gLFyf gsfi')))
    input_search.clear()
    input_search.send_keys(name)

    time.sleep(0.5)

    wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'gNO89b'))).click()

    time.sleep(0.5)

    # Extract the phone number here
    try:
        phone_number = ...
    except:
        pass

    driver.execute_script("window.history.go(-1)")

I should extract the phone number from the panel on the right of the page after clicking the button "Search" (if the panel is available).

Thanks in advance for your suggestions.

ADDENDUM

To find the phone numbers: inspecting the pages I have to search within the following span tags ('LrzXr zdqRlf kno-fv' is unique):

<span class="LrzXr zdqRlf kno-fv"><a data-dtype="d3ph" data-local-attribute="d3ph" jscontroller="LWZElb" href="#" jsdata="QKGTRc;_;B2Cx5o" jsaction="rcuQ6b:npT2md;F75qrd" data-ved="2ahUKEwik7rXAlNr2AhUFgP0HHedMA-IQkAgoAHoECDcQAw"><span><span aria-label="Chiama il numero di telefono 06 7047 6283">06 7047 6283</span></span></a></span>

and

<span class="LrzXr zdqRlf kno-fv"><a data-dtype="d3ph" data-local-attribute="d3ph" jscontroller="LWZElb" href="#" jsdata="QKGTRc;_;B4wf3Y" jsaction="rcuQ6b:npT2md;F75qrd" data-ved="2ahUKEwiwzs3blNr2AhU477sIHe5TA3sQkAgoAHoECEgQAw"><span><span aria-label="Chiama il numero di telefono 06 484467">06 484467</span></span></a></span>

My goal is to retrieve '06 7047 6283' and '06 484467'.

I succeeded using regular expressions but I would like to avoid them if possible:

content = wait.until(EC.visibility_of_element_located((By.XPATH, '//*[@id="kp-wp-tab-overview"]'))).text
phone_number = re.findall(r'Telefono: ([0-9 ]+)', content)[0]

Prophet · Accepted Answer · 2022-03-22 17:25:33Z

1

I'm not sure what is this line here for:

driver.execute_script('return document.querySelector("#L2AGLb > div")').click()

I can't see any element matching that locator there and it throws an exception for me.
Also I have fixed the locator of search input field and the way to click Enter there so my code is:

names = ['Ristorante Roma Antica', 'Ristorante e Braceria Al Piave']

for name in names:
    input_search = wait.until(EC.visibility_of_element_located((By.XPATH, "//input[@name='q']")))
    input_search.clear()
    input_search.send_keys(name + Keys.ENTER)

    phone = wait.until(EC.visibility_of_element_located((By.XPATH, "//span[contains(@aria-label,'Call phone number')]"))).text
    print(phone)
    driver.execute_script("window.history.go(-1)")

    time.sleep(0.5)

And it works.
At lest it opens the search results.

edited Mar 22, 2022 at 17:25

answered Mar 22, 2022 at 15:52

Prophet

33.5k29 gold badges58 silver badges90 bronze badges

Sign up to request clarification or add additional context in comments.

12 Comments

LJG Over a year ago

The line driver.execute_script('return document.querySelector("#L2AGLb > div")').click() is used to accept Google's terms of use when opening the page. If you are not logged in with a Google account, a form appears in which there is a button to click.

Prophet Over a year ago

I'm opening a new, clean session of Chrome without being logged into Google and I do not see any kind of pop-up of terms of use. But if you do see it - it's OK. You can continue with my code after accepting the terms of you you see in case this your code works for you

Prophet Over a year ago

Anyway, you can refer to the solution itself, without reference to driver.execute_script('return document.querySelector("#L2AGLb > div")').click() line, it was only a comment that this line was not relevant for me.

LJG Over a year ago

I'm trying to extract the phone number ("Telefono") from the results page (from the panel located to the right of the results page). If I use:

wait.until(EC.visibility_of_element_located((By.XPATH, '//*[@id="kp-wp-tab-overview"]/div[1]/div/div/div/div/div/div[8]/div/div/div/span[2]/span/a/span/span'))).text

I only find the first one. In your opinion is Google blocking me? (Note that I changed the list names)

LJG Over a year ago

I should get '06 7047 6283' and '06 484467'.

|

cruisepandey · Accepted Answer · 2022-03-22 17:14:27Z

There are few things in your code that can be improved:

for input use google input name q which has been there from last 5-6 years or may be even more.
You can directly send input, you don't need to explicitly have #wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'gNO89b'))).click()

Use the below XPath:

//a[text()='Phone']/../following-sibling::span/descendant::span[@aria-label]

to extract the phone numbers, based on Phone text.

Code:

names = ['Ristorante Roma Antica Roma', 'Ristorante e Braceria Al Piave Roma']

for name in names:
    input_search = wait.until(EC.visibility_of_element_located((By.NAME, 'q')))
    input_search.clear()
    input_search.send_keys(name, Keys.RETURN)

    time.sleep(0.5)

    #wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'gNO89b'))).click()

    #time.sleep(0.5)

    # Extract the phone number here
    try:
        phone_number = wait.until(EC.visibility_of_element_located((By.XPATH, "//a[text()='Phone']/../following-sibling::span/descendant::span[@aria-label]")))
        print(phone_number.text)
    except:
        pass

    driver.execute_script("window.history.go(-1)")

Imports:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

Output:

+39 06 7047 6283
+39 06 484467

Process finished with exit code 0

Collectives™ on Stack Overflow

Google search with Selenium Python doesn't work

2 Answers 2

12 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

12 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related