How to access HTML of webpage after button press

Question

I am trying to scrape some data from the following web page:

College Board - Georgia Institute of Technology

But the information I need to access is only displayed after pressing the "Applying" tab on the left. Since the URL does not change, how can I simulate pressing the button in order to scrape the HTML?

I am using Python3.3 and the requests module.

alecxe · Accepted Answer · 2014-03-16 04:43:11Z

1

According to the page source, the information you need is hidden inside a javascript code and is calculated and rendered after the click on "Applying" link.

requests simply cannot make in-browser user actions and, since there is no additional requests going after clicking "Applying", you cannot get the data without actually having a real browser to run that js code. Mechanize also wouldn't help because it cannot handle js.

Consider using selenium (FYI, you can also use a headless PhantomJS browser).

Hope that helps.

answered Mar 16, 2014 at 4:43

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to access HTML of webpage after button press

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related