transform JSON object into DataFrame on SPARK

Question

I'm getting a stream of tweets in JSON format using a Python notebook on Apache Spark on IBM Bluemix. I want to turn the JSON object into a dataframe. The closest thing i have found is:

json_obj = {"tweet": "hey man", "name": "Alan"} #mock example

tweetsDF = sqlContext.read.json("file path")

Now, I don't wanna load a file. I just wanna take that JSON variable and convert it into dataframe format. That way i can apply it to the tweets variable. Any idea how to do this? Thank you!

Christian Hirsch · Accepted Answer · 2016-01-19 01:33:48Z

6

First convert the JSON object into an RDD[String], then apply `sqlContext.read.json'. Here is example code in Scala

val json_obj = sc.parallelize(Array("""{"tweet": "hey man", "name": "Alan"}""", """{"tweet": "what's up", "name": "Bertha"}"""))
val tweetsDF = sqlContext.read.json(json_obj)
tweetsDF.show()
//+------+---------+
//|  name|    tweet|
//+------+---------+
//|  Alan|  hey man|
//|Bertha|what's up|
//+------+---------+

answered Jan 19, 2016 at 1:33

Christian Hirsch

2,06615 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

transform JSON object into DataFrame on SPARK

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related