Web Scraping javascript in Python / R

Question

I'm doing some personal data science projects and one of them is to see how often certain songs are played on the radio.

http://www.iheart.com/live/radio-1045-3401/

Looking at the above URL, when I look at page source, no values of interest populate. Not sure why, but when I use inspect element in chrome when I hover over the "Now Playing" header, I can see values for song and artist now playing.

Example:

a class="player-song" href="/artist/rem-3610/songs/-2450662/" title="Losing My Religion" data-reactid=".1hpdfx1l4ow.a.1.0.1.1">Losing My Religion</a

My two questions are:

Why isn't this showing up in page source, but I can see it under Inspect Element?
How would I web scrape this info since it is not appearing in page source?

haven´t checked but it´s probably content generated by javascript, in which case you could use selenium in python — juvian
– juvian, Commented Apr 8, 2016 at 18:16
you could probably also get the data using their API.. developer.iheart.com/Api/login — I wrestled a bear once.
– I wrestled a bear once., Commented Apr 8, 2016 at 18:28

Akshat Mahajan · Accepted Answer · 2016-04-09 22:08:45Z

3

Most web pages that involve dynamic elements have page elements generated and inserted by Javascript that the browser parses and executes for you. You already guessed this, I suspect, based on the question title.

What you see in the page source is the raw HTML before Javascript kicks in and updates it.
You want a headless browser: a browser without a graphical user interface. This will parse and execute Javascript for you, and update page HTML accordingly.

Here is a full list of headless browsers. Note that you can do this task in any language.

edited Apr 9, 2016 at 22:08

answered Apr 8, 2016 at 18:19

Akshat Mahajan

9,8764 gold badges38 silver badges46 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Ehvince Over a year ago

MechanicalSoup states in their readme that it doesn't do javascript, and RoboBrowser is a mix of requests and beautifulSoup so I think not. Not sure about Mechanize too. Nice answer though.

Collectives™ on Stack Overflow

Web Scraping javascript in Python / R

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related