How to read in-memory JSON string into Spark DataFrame

Question

I'm trying to read an in-memory JSON string into a Spark DataFrame on the fly:

var someJSON : String = getJSONSomehow()
val someDF : DataFrame = magic.convert(someJSON)

I've spent quite a bit of time looking at the Spark API, and the best I can find is to use a sqlContext like so:

var someJSON : String = getJSONSomehow()
val tmpFile : Output = Resource
    .fromFile(s"/tmp/json/${UUID.randomUUID().toString()}")
tmpFile.write("hello")(Codec.UTF8)
val someDF : DataFrame = sqlContext.read().json(tmpFile)

But this feels kind of awkward/wonky and imposes the following constraints:

It requires me to format my JSON to one object per line (per documentation); and
It forces me to write the JSON to a temp file, which is slow and awkward; and
It forces me to clean up temp files over time, which is cumbersome and feels "wrong" to me

So I ask: Is there a direct and more efficient way to convert a JSON string into a Spark DataFrame?

Possible duplicate of how to convert json string to dataframe on spark — cheseaux
– cheseaux, Commented Sep 21, 2016 at 15:46

bear911 · Accepted Answer · 2016-09-21 14:52:50Z

14

From Spark SQL guide:

val otherPeopleRDD = spark.sparkContext.makeRDD(
"""{"name":"Yin","address":{"city":"Columbus","state":"Ohio"}}""" :: Nil)
val otherPeople = spark.read.json(otherPeopleRDD)
otherPeople.show()

This creates a DataFrame from an intermediate RDD (created by passing a String).

answered Sep 21, 2016 at 14:52

bear911

3492 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Thomas Decaux Over a year ago

The very good thing is, you can use it to filter wrong line before parsing (using sqlContext.read.json(sc.textFile("...").filter(....)))

Collectives™ on Stack Overflow

How to read in-memory JSON string into Spark DataFrame

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related