Simulate browser execution to execute a script

Question

I want to crawl the data at the bottom of this webpage (the historical_table). If I use requests.get it returns the source code of the page, that contains only the script to create the table. The only way I know to crawl it is using selenium.webdriver (so via a browser session); however, this method is slower.
Is there a way to simulate the browser execution of the script without opening a browser session at all?

Possible duplicate of stackoverflow.com/questions/7593611/… check this — Vivek Kalyanarangan
– Vivek Kalyanarangan, Commented Mar 2, 2017 at 8:23
No, I'd like to use another tool, not selenium. I'd like to use a requests-like process that gets the HTML of the page AND execute the script. — Matteo Felici
– Matteo Felici, Commented Mar 2, 2017 at 8:26

Tamas Hegedus · Accepted Answer · 2017-03-02 08:26:56Z

1

It is possible but it means you are writing your own browser which I dont recommend. Use a headless browser instead, like http://phantomjs.org . These are meant to be used this way.

answered Mar 2, 2017 at 8:26

Tamas Hegedus

30.1k12 gold badges67 silver badges102 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Simulate browser execution to execute a script

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related