How to read in JSON file and write as JSON Lines file in Spark Scala?

Question

I have a JSON file that I'd like to convert to JSON Lines in spark scala. I was able to figure it out in Python by just using Pandas read_json method and then writing it with some special lines parameters passed in.

Say the original format is:

{
            "A": "400",
            "B": "100",
            "C": "DEM",
            "D": "USD",
            "E": "80029898",
            "F": "1.64110-",
            "G": "0 "

        },
        {
            "A": "400",
            "B": "100",
            "C": "USD",
            "D": "DEM",
            "E": "80029898",
            "F": "1.64110 ",
            "G": "0 ",

        },

I'd like to write is as:

{"A":"400","B":"100","C":"DEM","D":"USD","E":"80029898","F":"1.64110-","G":"0"}
{"A":"400","B":"100","C":"USD","D":"DEM","E":"80029898","F":"1.64110 ","G":"0"}

Thanks so much and have a great day!

The input format is a comma separated list of JSONs or should there be brackets [...] ? — Nebril
– Nebril, Commented Jan 26, 2021 at 16:24
JSONs can be read using spark.read.json(), and written using df.write.json(). Refer this link for more info and to build on your case — samkart
– samkart, Commented Jan 26, 2021 at 17:04

Amardeep Flora · Accepted Answer · 2021-01-27 07:10:53Z

1

If you are using > spark 2.2, you can use :

spark.read
  .option("multiLine", true).option("mode", "PERMISSIVE")
  .json("/path/to/user.json")

and then use to write it in desired format:

df.write.json()

answered Jan 27, 2021 at 7:10

Amardeep Flora

1,3806 gold badges14 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to read in JSON file and write as JSON Lines file in Spark Scala?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related