Trying to read a json file into a Map[String,Object] in scala

Question

I'm trying to read out a json config file into my scala project. The format of the json is as follows:

{
  "parameters": [
    {
      "name": "testInteger",
      "type": "Integer",
      "value": "10"
    },
    {
      "name": "testString",
      "type": "String",
      "value": "yeah"
    }
  ]
}

I have been using spark to generate a data frame

val df = spark.read.option("multiline","true").json(path)

I am needing the data from the json file to then be read into a Map that has key "name" and value of the specified type

Expected Output:

Map: "testInteger" -> 10
     "testString" -> "yeah"

I am new to scala and unsure where to start, any advice would be appreciated.

(Note: using Java 8 and intellij to write)

Ideally would like to be "testInteger" -> 10, "testString" -> "yeah" — needsomehelp
– needsomehelp, Commented May 22, 2020 at 5:15
I should clarify, there is potentially going to be more than one Map object. In this case the map would be called Parameters and would map a string (name) to the value of the specified type. I need to have the functionality to expand to include more maps — needsomehelp
– needsomehelp, Commented May 22, 2020 at 5:18

Chitral Verma · Accepted Answer · 2020-05-22 06:16:13Z

So, this is what you should do,

Create SparkSession,

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types.{ArrayType, StructType}

val spark = SparkSession.builder().master("local[2]").getOrCreate()
import spark.implicits._

Create Schema,

val schema = new StructType().add(
"parameters",ArrayType.apply(
      new StructType()
          .add("name", "string")
          .add("type", "string")
          .add("value", "string")
       ))

Read data set,

 val df = spark.read
      .option("multiline", "true")
      .schema(schema)
      .json("/path/to/json")
      .select(explode(col("parameters")).alias("params"))

This will give you a struct column called 'params' with fields name, type and value. This will look like,

root
 |-- params: struct (nullable = true)
 |    |-- name: string (nullable = true)
 |    |-- type: string (nullable = true)
 |    |-- value: string (nullable = true)

Note: All struct and map type columns impose type safety. So the schema cannot allow values of a different type in the same column. Thus all your values in value field will be cast to string. As per your use case, you can use udf to cast things at runtime using the field type.

Collectives™ on Stack Overflow

Trying to read a json file into a Map[String,Object] in scala

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related