0

I have 4 json files spread into to folders: folder1 and folder2. Each json file contains the date, the body and the title.

folder1.json:

{"date": "December 31, 1989, Sunday, Late Edition - Final", "body": "Frigid temperatures across much of the United States this month sent demand for heating oil soaring, providing a final upward jolt to crude oil prices. That's assuming the economy performs as expected - about 1 percent growth in G.N.P. The other big uncertainty is the U.S.S.R. If their production drops more than 4 percent, prices could stengthen. ", "title": "Prospects;"}

{"date": "December 31, 1989, Sunday, Late Edition - Final", "body": "DATELINE: WASHINGTON, Dec. 30 For years, experts have dubbed Czechoslovakia's spy agency the ''two Czech'' service. Agents of the Office for the Protection of State Secrets got one check from Prague, the pun goes, and another from their real bosses at K.G.B. headquarters in Moscow. Roy Godson, head of the Washington-based National Strategy Information Center and a well-known intelligence scholar, called any democratic change ''a net loss'' for Soviet intelligence. But he cautioned against euphoria. ''The Soviets wouldn't have relied on just official cooperation,'' he said. ''It would be surprising if they haven't unilaterally penetrated friendly services with their own agents, too.'' ", "title": "Upheaval in the East: Espionage;"}

folder2.json:

{"date": "December 31, 1989, Sunday, Late Edition - Final", "body": "SURVIVING the decline in the economy will be the overriding issue for 1990, say leaders of the county's business community.  But facing business owners are numerous problems, from taxes and regulations at all levels of government to competition from other businesses in and out of Westchester. Successful Westchester business owners will face and overcome these risks and obstacles. Westchester is a land of opportunity for the business owner. ", "title": "Coping With the Economic Prospects of 1990"}

{"date": "December 29, 1989, Friday, Late Edition - Final", "body": "Eastern Airlines said yesterday that it was laying off 600 employees, mostly managers, and cutting wages by 10 percent or 20 percent for about half its work force. Thomas J. Matthews, Eastern's senior vice president of human resources, estimated that the measures would save the carrier about $100 million a year. Eastern plans to rebuild by making Atlanta its primary hub and expects to operate about 75 percent of its flights from there. ", "title": "Eastern Plans Wage Cuts, 600 Layoffs"}

I will like to create a common list of all these json files but only with the body of each article. So far I am trying the following:

json1 <- lapply(readLines("folder1.json"), fromJSON)
json2 <- lapply(readLines("folder2.json"), fromJSON)

jsonl <- list(json1$body, json2$body)

But it is not working. Any suggestions?

Improve this question
1
  • 1
    ?purrr::map shld provide inspiration Commented Sep 26, 2016 at 11:11

2 Answers 2

Reset to default
1

Andres Azqueta

Solution: You need to derence the the fromJSON(), in the sapply() to only retrieve the body.

  • fromJSON()$body

Note: I am assuming the file format from you previous question

The point being the file format is sudo JSON, hence the modified fromJSON() call below.

OK, Let step through an example:

Stage 1: Concatenate JSON files into 1

filelist <- c("./data/NYT_1989.json", "./data/NYT_1990.json")
newJSON <- sapply(filelist, function(x) fromJSON(sprintf("[%s]", paste(readLines(x), collapse = ",")), flatten = FALSE))
newJSON[2]# Extract bodies
newJSON[5]# Extract bodies

Output

filelist <- c("./data/NYT_1989.json", "./data/NYT_1990.json")
> newJSON <- sapply(filelist, function(x) fromJSON(sprintf("[%s]", paste(readLines(x), collapse = ",")), flatten = FALSE))
> newJSON[2]# Extract bodies
[[1]]
[1] "Frigid temperatures across much of the United States this month sent demand for heating oil soaring, providing a final upward jolt to crude oil prices. Some spot crude traded at prices up 40 percent or more from a year ago. Will these prices hold? Five experts on oil offer their views. That's assuming the economy performs as expected - about 1 percent growth in G.N.P. The other big uncertainty is the U.S.S.R. If their production drops more than 4 percent, prices could stengthen. "
[2] "DATELINE: WASHINGTON, Dec. 30 For years, experts have dubbed Czechoslovakia's spy agency the ''two Czech'' service. But he cautioned against euphoria. ''The Soviets wouldn't have relied on just official cooperation,'' he said. ''It would be surprising if they haven't unilaterally penetrated friendly services with their own agents, too.'' "                                                                                                                                                
[3] "SURVIVING the decline in the economy will be the overriding issue for 1990, say leaders of the county's business community. Successful Westchester business owners will face and overcome these risks and obstacles. Westchester is a land of opportunity for the business owner. "                                                                                                                                                                                                                  

> newJSON[5]# Extract bodies
[[1]]
[1] "Blue temperatures across much of the United States this month sent demand for heating oil soaring, providing a final upward jolt to crude oil prices. Some spot crude traded at prices up 40 percent or more from a year ago. Will these prices hold? Five experts on oil offer their views. That's assuming the economy performs as expected - about 1 percent growth in G.N.P. The other big uncertainty is the U.S.S.R. If their production drops more than 4 percent, prices could stengthen. "
[2] "BLUE1: WASHINGTON, Dec. 30 For years, experts have dubbed Czechoslovakia's spy agency the ''two Czech'' service. But he cautioned against euphoria. ''The Soviets wouldn't have relied on just official cooperation,'' he said. ''It would be surprising if they haven't unilaterally penetrated friendly services with their own agents, too.'' "                                                                                                                                                 
[3] "GREEN4 the decline in the economy will be the overriding issue for 1990, say leaders of the county's business community. Successful Westchester business owners will face and overcome these risks and obstacles. Westchester is a land of opportunity for the business owner. "

Stage 2: Concatenate and extract the body from all files...

Look for the reference to fromJSON()$body in code line...

filelist <- c("./data/NYT_1989.json", "./data/NYT_1990.json")
newJSON <- sapply(filelist, function(x) fromJSON(sprintf("[%s]", paste(readLines(x), collapse = ",")), flatten = FALSE)$body)
newJSON

Output

> filelist <- c("./data/NYT_1989.json", "./data/NYT_1990.json")
> newJSON <- sapply(filelist, function(x) fromJSON(sprintf("[%s]", paste(readLines(x), collapse = ",")), flatten = FALSE)$body)
> newJSON
     ./data/NYT_1989.json                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
[1,] "Frigid temperatures across much of the United States this month sent demand for heating oil soaring, providing a final upward jolt to crude oil prices. Some spot crude traded at prices up 40 percent or more from a year ago. Will these prices hold? Five experts on oil offer their views. That's assuming the economy performs as expected - about 1 percent growth in G.N.P. The other big uncertainty is the U.S.S.R. If their production drops more than 4 percent, prices could stengthen. "
[2,] "DATELINE: WASHINGTON, Dec. 30 For years, experts have dubbed Czechoslovakia's spy agency the ''two Czech'' service. But he cautioned against euphoria. ''The Soviets wouldn't have relied on just official cooperation,'' he said. ''It would be surprising if they haven't unilaterally penetrated friendly services with their own agents, too.'' "                                                                                                                                                
[3,] "SURVIVING the decline in the economy will be the overriding issue for 1990, say leaders of the county's business community. Successful Westchester business owners will face and overcome these risks and obstacles. Westchester is a land of opportunity for the business owner. "                                                                                                                                                                                                                  
     ./data/NYT_1990.json                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
[1,] "Blue temperatures across much of the United States this month sent demand for heating oil soaring, providing a final upward jolt to crude oil prices. Some spot crude traded at prices up 40 percent or more from a year ago. Will these prices hold? Five experts on oil offer their views. That's assuming the economy performs as expected - about 1 percent growth in G.N.P. The other big uncertainty is the U.S.S.R. If their production drops more than 4 percent, prices could stengthen. "
[2,] "BLUE1: WASHINGTON, Dec. 30 For years, experts have dubbed Czechoslovakia's spy agency the ''two Czech'' service. But he cautioned against euphoria. ''The Soviets wouldn't have relied on just official cooperation,'' he said. ''It would be surprising if they haven't unilaterally penetrated friendly services with their own agents, too.'' "                                                                                                                                                 
[3,] "GREEN4 the decline in the economy will be the overriding issue for 1990, say leaders of the county's business community. Successful Westchester business owners will face and overcome these risks and obstacles. Westchester is a land of opportunity for the business owner. "
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1 
require(RJSONIO)   
json_1<- fromJSON("~/folder1/1.json")
json_2<- fromJSON("~/folder2/2.json")

jsonl <- list(json1$body, json2$body)
Improve this answer

1 Comment

library(jsonlite) json_1<- stream_in(file("~/folder1/1.json")) json_2<- stream_in(file("~/folder/2.json")) jsonl <- list(json1$body, json2$body)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.