Using python how do I click a button to scrape the content hidden

Question

Hello I wrote a function using selenium to click the "advisor" button so that I can scrape the table hidden. when I run it my chrome driver successfully opens and visits the page.. but the button does not get clicked. I hope, you guys help me to figure this out? NOTE: I am new to scraping techniques. also please let me know If this can be done with bs4. Here is a code:

def scrapper():
    u = "https://teqatlas.com/products-and-services/0chain"
    browser = webdriver.Chrome(executable_path=binary_path)
    wait = WebDriverWait(browser, 10)
    browser.set_page_load_timeout(10)
    # stop load after a timeout
    try:
        browser.get(u)
    except TimeoutException:
        browser.execute_script("window.stop();")
        
    button = browser.find_element_by_xpath('//button[@class="o5ph61-3 eBqrHG"]')
    if button:
        button.click()

scrapper()

αԋɱҽԃ αмєяιcαη · Accepted Answer · 2020-03-31 08:55:36Z

2

from selenium import webdriver
import pandas as pd
from selenium.webdriver.firefox.options import Options

options = Options()
options.add_argument('--headless')
driver = webdriver.Firefox(options=options)

driver.get("https://teqatlas.com/products-and-services/0chain")

btn = driver.find_element_by_css_selector("button.o5ph61-3.faMQuX").click()
df = pd.read_html(driver.page_source)[0]

df.to_csv("data.csv", index=False)

driver.quit()

Output: view-online

answered Mar 31, 2020 at 8:55

αԋɱҽԃ αмєяιcαη

11.6k3 gold badges23 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Sabbir Talukdar Over a year ago

let me install firefox, sorry

αԋɱҽԃ αмєяιcαη Over a year ago

@SabbirTalukdar you can use Chrome, Edge, Firefox and Safari. check docs to download your preferred driver.

Sabbir Talukdar Over a year ago

WebDriverException: Message: 'geckodriver' executable needs to be in PATH.

αԋɱҽԃ αмєяιcαη Over a year ago

@SabbirTalukdar you need to put geckodriver within your Python installation folder.

Sameer Arora · Accepted Answer · 2020-03-31 08:51:40Z

0

The xpath that you are using is incorrect. Please pick the below xpath to click on the button:

WebDriverWait(browser, 20).until(EC.presence_of_element_located((By.XPATH, "//button[text()='Advisories']"))).click()

You need to add the following imports:

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

edited Mar 31, 2020 at 8:51

answered Mar 31, 2020 at 8:30

Sameer Arora

4,5073 gold badges12 silver badges20 bronze badges

2 Comments

Sabbir Talukdar Over a year ago

thank you so much, I will implement the code and will loet you know the result

Sameer Arora Over a year ago

@SabbirTalukdar please pick the updated ans, i have added explicit wait in this.

Madhur Gupta · Accepted Answer · 2020-03-31 09:21:36Z

0

Does this work? I am also new to selenium. I changed find_element_by_xpath to find_element_by_class_name here though.

from selenium.webdriver.common.action_chains import ActionChains
def business_description_scrapper():
    u = "https://teqatlas.com/products-and-services/0chain"
    browser = webdriver.Chrome(executable_path=binary_path)
    wait = WebDriverWait(browser, 10)
    browser.set_page_load_timeout(10)
    # stop load after a timeout
    try:
        browser.get(u)
    except TimeoutException:
        browser.execute_script("window.stop();")

    button = browser.find_element_by_class_name("o5ph61-3.eBqrHG")
    if button:
        actions = ActionChains(browser)
        actions.click(button).perform()

business_description_scrapper()

I tested it and the button got clicked.

edited Mar 31, 2020 at 9:21

answered Mar 31, 2020 at 8:28

Madhur Gupta

156 bronze badges

7 Comments

Sabbir Talukdar Over a year ago

thank you so much, I will implement the code and will loet you know the result

Madhur Gupta Over a year ago

Does it find the button and unable to click? Or unable to find button?

Sabbir Talukdar Over a year ago

a button got clicked but it was not the "advisories" button..

Madhur Gupta Over a year ago

@Sabbir How about now?

Sabbir Talukdar Over a year ago

this is the most troubling thing.. no error messages show up

|

Collectives™ on Stack Overflow

Using python how do I click a button to scrape the content hidden

3 Answers 3

4 Comments

2 Comments

7 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

2 Comments

7 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related