I'm really new to web scraping. Is there anyone that could tell me how to search on google.com with Selenium in Python?
3 Answers
In order to search something on google, try this:
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('https://www.google.com/')
textbox = driver.find_element_by_xpath('//input[@name=\"q\"]')
textbox.send_keys('who invented python\n')
1 Comment
chronovirus
Hey, @Sushil I tried the code but this error occoured:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[@id="tsf"]/div[2]/div[1]/div[1]/div/div[2]/input"} It might be something with the fact that the Cookie Alert pops up?Selenium probably isn't the best. other libraries/tools would work better. BeautifulSoup is the first one that comes to mind
1 Comment
chronovirus
Thank you I'll look into that
Don't use Selenium, there are other tools more suitable for that. Be aware that Google does not look upon favorably on scraping their search results:
Google does not take legal action against scraping, likely for self-protective reasons. However, Google is using a range of defensive methods that makes scraping their results a challenging task.
Source: https://en.wikipedia.org/wiki/Search_engine_scraping
2 Comments
chronovirus
What other tools would you recommend?
John Donn
I never used it, but github.com/scrapy/scrapy, dzone.com/articles/getting-started-with-scrapy looks worthwile to try. But also BeautiulSoup should be OK.