How to read multiple Json files under sub directories using Scala

Question

I am looking a code snippet to find the best practice to read multiple nested JSON files under sub directories in hadoop using scala .

If we can write into one single file in some other directory in hadoop the above JSON files , that would be even better .

Any help is appreciated.

Thanks PG

: are you using Spark with Scala API or how you are using Scala in Hadoop? — Shankar
– Shankar, Commented Sep 29, 2016 at 6:44
You can use sqlContext.read.json("json file path") to read json file, it returns an DataFrame. But you said nested directories, is the json files are having different schemas? — Shankar
– Shankar, Commented Sep 29, 2016 at 14:38
Thanks Shankar . Files will be of similar schemas , and I guess it worked to read the files. Now next step is can I write all the files into one single json file may be in 1-2 steps to be performance efficient. — user3054752
– user3054752, Commented Sep 29, 2016 at 20:20
Take a look here. I think the top answer may help: stackoverflow.com/questions/28203217/… — sascha10000
– sascha10000, Commented Sep 29, 2016 at 23:31

Shankar · Accepted Answer · 2016-09-30 08:52:29Z

0

You can use sqlContext.read.json("input file path") to read json file, it returns an DataFrame.

Once you got the DataFrame, just use df.write.json("output file path") to write the DF as json file.

Code example: if you use Spark 2.0

val spark = SparkSession
      .builder()
      .appName("Spark SQL JSON example")
      .getOrCreate()

      val df = spark.read.json("input/file/path")

      df.write.json("output/file/path")

answered Sep 30, 2016 at 8:52

Shankar

9,02926 gold badges100 silver badges172 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to read multiple Json files under sub directories using Scala

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related