Click button on website then scrape web page

Question

I have a website I would like to click a button on then scrape the website using python the html code between the button is:

 <span id="exchange-testing" class="exchange-input nav-link" data track="&amp;lid=testing&amp;lpos=site_settings" data-value="testing">Testing</span>

Is this possible? I am able to scrape all the data I need from the page but I need to click the button first.

Any help would be appreciated

alecxe · Accepted Answer · 2014-11-09 00:38:21Z

26

Basically, you have two options:

high-level approach: automate a real browser using selenium or, in other words, make the browser repeat all the user actions needed to get to the page with the desired data.
low-level approach: when you click the button, investigate what is happening under the hood - explore the "Network" tab of browser developer tools and see what requests are being made. Then, simulate them in your scraper. Here, you may consider using tools like requests, mechanize for making requests, handling scraping sessions, submitting forms etc and tools like BeautifulSoup, lxml.html for html parsing. Also, Scrapy web-scraping framework is a must see.

answered Nov 9, 2014 at 0:32

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

rene · Accepted Answer · 2020-05-09 09:02:43Z

1

If there is a button you want to click then scrape then do the following:

Inspect the button element . the link the button will open after clicking it will always be in the html you opened .
Scrape this link and do requests.get(link) . this becomes equivalent to clicking button .

but this isn't applicable when the button needs to activate some js function that should've run on clicking the element.

rene

42.6k78 gold badges122 silver badges166 bronze badges

answered May 9, 2020 at 7:09

TonyMontana

2617 silver badges19 bronze badges