Reading multiple JSON files in a directory into one Data Frame

Question

library(rjson)
filenames <- list.files(pattern="*.json") # gives a character vector, with each file name represented by an entry

Now I want to import all the JSON files into R as one single dataFrame. How do I do that?

I first tried

myJSON <- lapply(filenames, function(x) fromJSON(file=x)) # should return a list in which each element is one of the JSON files

but the above code takes along time to terminate, since I have 15,000 files, and I know it won't return a single data frame. Is there a faster way to do this?

Sample JSON file:

 {"Reviews": [{"Ratings": {"Service": "4", "Cleanliness": "5"}, "AuthorLocation": "Boston", "Title": "\u201cExcellent Hotel & Location\u201d", "Author": "gowharr32", "ReviewID": "UR126946257", "Content": "We enjoyed the Best Western Pioneer Square....", "Date": "March 29, 2012"}, {"Ratings": {"Overall": "5"},"AuthorLocation": "Chicago",....},{...},....}]}

if/when it returns, you can call do.call(rbind,myJSON) to get a single data.frame, provided the data.frame's are compatible... — Jthorpe
– Jthorpe, Commented Feb 16, 2016 at 1:02
a sample of one of the JSON files would go a long way in helping folks to help you. purrr::map_df or purrr::flatten_df could help. pbapply::pblapply with dplyr::bind_rows wld give you a progress bar if you could get the fromJSON to give back a data.frame — hrbrmstr
– hrbrmstr, Commented Feb 16, 2016 at 1:28

Monduiz · Accepted Answer · 2018-01-03 01:04:10Z

15

For anyone looking for a purrr / tidyverse solution coming here:

library(purrr)
library(tidyverse)
library(jsonlite)

path <- "./your_path"
files <- dir(path, pattern = "*.json")

data <- files %>%
       map_df(~fromJSON(file.path(path, .), flatten = TRUE))

answered Jan 3, 2018 at 1:04

Monduiz

72110 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Geek On Acid · Accepted Answer · 2017-07-25 22:14:15Z

3

Go parallel via:

library(parallel)
cl <- makeCluster(detectCores() - 1)
json_files<-list.files(path ="your/json/path",pattern="*.json",full.names = TRUE)
json_list<-parLapply(cl,json_files,function(x) rjson::fromJSON(file=x,method = "R"))
stopCluster(cl)

edited Jul 25, 2017 at 22:14

Geek On Acid

6,4104 gold badges48 silver badges64 bronze badges

answered Feb 28, 2017 at 15:25

amonk

1,7973 gold badges18 silver badges29 bronze badges

Collectives™ on Stack Overflow

Reading multiple JSON files in a directory into one Data Frame

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related