Web scraping and parsing HTML in R

Question

I'm trying to parse this webpage into a dataframe but keep getting stuck using the XML package being told it's not XML.

I would like to take the below text and convert into a table/data.frame - what is the easiest way to do this after i've taken the URL text and htmlParsed it?

doc = getURL("http://m.racingpost.com/card/blocks.sd?race_id=first&r_date=2015-03-28&tab=card&view=meetings&blocks=cards-list&_=1427439140572") doc = htmlParse(doc, asText=T)

jdharrison · Accepted Answer · 2015-03-28 08:23:08Z

3

The URL is returning JSON. You can parse it using a number of R packages RJSONIO, rjson and jsonlite:

library(jsonlite)
appURL <- "http://m.racingpost.com/card/blocks.sd?race_id=first&r_date=2015-03-28&tab=card&view=meetings&blocks=cards-list&_=1427439140572"
appDATA <- fromJSON(appURL)
appITEMS <- appDATA[["cards-list"]][["items"]]
> appITEMS$c1083
$abandonedCount
[1] 0

$crsName
[1] "Chelmsford (AW)"

$crsAbbr
[1] "Cfd"

$isForeign
[1] ""

$races
id                                                           title distance cls crsId time       date
1 620151        Buy Online At chelmsfordcityracecourse.com Maiden Stakes       1m   4  1083 2:20 2015-03-28
2 620152 Dubai World Cup toteplacepot Today Maiden Stakes (Plus 10 Race)       5f   4  1083 2:55 2015-03-28
3 620153                            &pound;1 Million totescoop6 Handicap       5f   2  1083 3:30 2015-03-28
4 620154                                toteexacta Pick The 1,2 Handicap       6f   4  1083 4:05 2015-03-28
5 620155               totetrifecta Pick The 1,2,3 Handicap (Bobis Race)       1m   3  1083 4:40 2015-03-28
6 620156                                               totepool Handicap     1m2f   2  1083 5:15 2015-03-28
7 620157                                  Madness Live 3rd June Handicap     1m2f   4  1083 5:50 2015-03-28
timestamp raceGroup hCount abandoned videoId    going offers
1 1427552400                8             57049 Standard   NULL
2 1427554500                5             57050 Standard   NULL
3 1427556600  Handicap     12             57051 Standard   NULL
4 1427558700  Handicap      7             57052 Standard   NULL
5 1427560800  Handicap      8             57053 Standard   NULL
6 1427562900  Handicap      7             57054 Standard   NULL
7 1427565000  Handicap      6             57055 Standard   NULL

The data is not returned in a tabular format but you can work with the individual "items" to fit your needs. The jsonlite package helpfully returns appropriate tabular structures also.

answered Mar 28, 2015 at 8:23

jdharrison

30.5k4 gold badges83 silver badges91 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Tammboy Over a year ago

That's looks great but when i do it i get this error:

Tammboy Over a year ago

Error in feed_push_parser(buf) : lexical error: invalid char in json text. <html><head><title>Request Reje (right here) ------^

Tammboy Over a year ago

When i installed the curl package it told me it was for an earlier version of r - is this the difference?

Tammboy Over a year ago

Managed to fix that by using getURL to bring in the json text first - seem to solve it - thanks

Collectives™ on Stack Overflow

Web scraping and parsing HTML in R

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related