R: Web scraping JSON, extracting information from nest

Question

I am trying to use tidyJSON to extract information from JSON, but I am open to any R package that can achieve my ends. I took a look at the documentation and vignittes and found the complex example was helpful. However, the information I want is nested inside of a non-key-value pair and I am not sure how to access it. I am interested in getting appid, name, developer, etc., but this information is within 570 and 730:

{"570":{"appid":570,"name":"Dota 2","developer":"Valve","publisher":"Valve","score_rank":71,"owners":102151578,"owners_variance":259003,"players_forever":102151578,"players_forever_variance":259003,"players_2weeks":9436299,"players_2weeks_variance":89979,"average_forever":11727,"average_2weeks":1229,"median_forever":277,"median_2weeks":662,"ccu":811259,"price":"0","tags":{"Free to Play":22678,"MOBA":7808,"Strategy":7415,"Multiplayer":6757,"Team-Based":4848,"Action":4602,"e-sports":4089,"Online Co-Op":3669,"Competitive":3553,"PvP":2655,"RTS":2267,"Difficult":2129,"RPG":2114,"Fantasy":2044,"Tower Defense":2024,"Co-op":1898,"Character Customization":1514,"Replay Value":1487,"Action RPG":1397,"Simulation":1024}},

"730":{"appid":730,"name":"Counter-Strike: Global Offensive","developer":"Valve","publisher":"Valve","score_rank":78,"owners":29225079,"owners_variance":154335,"players_forever":28552354,"players_forever_variance":152685,"players_2weeks":9102348,"players_2weeks_variance":88410,"average_forever":17648,"average_2weeks":791,"median_forever":5030,"median_2weeks":358,"ccu":543626,"price":"1499","tags":{"FPS":17082,"Multiplayer":13744,"Shooter":12833,"Action":10881,"Team-Based":10369,"Competitive":9664,"Tactical":8529,"First-Person":7329,"e-sports":6716,"PvP":6383,"Online Co-Op":5714,"Military":4621,"Co-op":4435,"Strategy":4424,"War":4361,"Realistic":3196,"Trading":3191,"Difficult":3158,"Fast-Paced":3100,"Moddable":2496}}

There are many thousands of such entries. Is there a way to skip the "top-level" and look within the nest?
The JSON information is from http://steamspy.com/api.php?request=top100in2weeks

You can try listviewer::jsonedit to help you visualize the data first. The maybe jsonlite couçd help you extract what you need. — RobertMyles
– RobertMyles, Commented May 25, 2017 at 18:09

R. Schifini · Accepted Answer · 2017-05-26 01:54:53Z

1

This might be what you need:

library(jsonlite)
data = fromJSON("http://steamspy.com/api.php?request=top100in2weeks")

appid = lapply(data, function(x){x$appid})
name = lapply(data, function(x){x$name})

df = data.frame(appid = unlist(appid),
                name = unlist(name),
                stringsAsFactors = F)

Result:

> head(df)
        appid                             name
570       570                           Dota 2
730       730 Counter-Strike: Global Offensive
578080 578080    PLAYERUNKNOWN'S BATTLEGROUNDS
440       440                  Team Fortress 2
271590 271590               Grand Theft Auto V
433850 433850           H1Z1: King of the Kill

I'll let you add the rest of the information

Edit: Adding arrays to a dataframe

Adding the tags information for each game in the data frame is possible. And the times tagged as well. For each game you must store an array of tag names in a column and the tag quantities in another.

After the definition of df add the following lines:

for(k in 1:nrow(d)){
    d$tags[k] = list(names(data[[k]]$tags))
    d$tagsQ[k] = list(unlist(data[[k]]$tags))
}

This will give you:

> d["570",]
    appid   name
570   570 Dota 2

tags
570 Free to Play, MOBA, Strategy, Multiplayer, Team-Based, Action, e-sports, Online Co-Op, Competitive, PvP, RTS, Difficult, RPG, Fantasy, Tower Defense, Co-op, Character Customization, Replay Value, Action RPG, Simulation

tagsQ
570 22686, 7810, 7420, 6759, 4850, 4603, 4092, 3672, 3555, 2657, 2267, 2130, 2116, 2045, 2024, 1898, 1514, 1487, 1397, 1023

In this situation, columns tags and tagsQ contain lists. To obtain the second tag and quantity for appid 570 do:

> df["570","tags"][[1]][2]
[1] "MOBA"

> d["570","tagsQ"][[1]][2]
MOBA 
7810

edited May 26, 2017 at 1:54

answered May 25, 2017 at 18:08

R. Schifini

9,3532 gold badges31 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user2205916 Over a year ago

thanks. I am also struggling with converting "tags" field into a data structure that can be put into the data frame. I end up with a named list that can't be inserted into the data frame. Is there an easy way to convert the tags to dummy boolean columns in the data frame, or to concatenate it into a comma-separated value in a field of the data frame? I am really bad with list structures.

user2205916 Over a year ago

I tried: r-tutor.com/r-introduction/list/named-list-members, and stackoverflow.com/questions/32059798/… and Google

Collectives™ on Stack Overflow

R: Web scraping JSON, extracting information from nest

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related