Extract a Json from an array inside a json in spark

Question

I have a complicated JSON column whose structure is :

story{ cards: [{story-elements: [{...}{...}{...}}]}

The length of the story-elements is variable. I need to extract a particular JSON block from the story-elements array. For this, I first need to extract the story-elements.

Here is the code which I have tried, but it is giving error:

import org.json4s.{DefaultFormats, MappingException}
import org.json4s.jackson.JsonMethods._
import org.apache.spark.sql.functions._

def getJsonContent(jsonstring: String): (String) = {
implicit val formats = DefaultFormats
val parsedJson = parse(jsonstring)
val value1 = (parsedJson\"cards"\"story-elements").extract[String]
value1
}
val getJsonContentUDF = udf((jsonstring: String) => 
getJsonContent(jsonstring))

input.withColumn("cards",getJsonContentUDF(input("storyDataFrame")))

{ story: { cards: [ { story-elements: [ { description: "", page-url: "/story/03d952fb-83f9-468e-a02c-fe6e2431fa6c/element/fa52a7d9-1d02-4214-b7ab-e86c8c635403", type: "text", family-id: "22f2d777-d51e-435a-a4d8-ba47f0b9e257", title: "", id: "fa52a7d9-1d02-4214-b7ab-e86c8c635403", metadata: { }, subtype: null, text: "<p>The tragic death of the two Kannada actors Raghava Uday and Anil on Monday</p>" } ]} ] } — NamrataK
– NamrataK, Commented Jul 5, 2018 at 11:25

Eugene Platonov · Accepted Answer · 2018-07-05 21:50:39Z

0

According to json you provided, story-elements is a an array of json objects, but you trying to extract array as a string ((parsedJson\"cards"\"story-elements").extract[String]).

You can create case class representing on story (like case class Story(description: String, pageUrl: String, ...)) and then instead of extract[String], try extract[List[Story]] or extract[Array[Story]] If you need just one piece of data from story (e.g. descrition), then you can use xpath-like syntax to get that and then extract List[String]

answered Jul 5, 2018 at 21:50

Eugene Platonov

3,3353 gold badges29 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Extract a Json from an array inside a json in spark

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related