Consider the code:
.withColumn("my_column",
aggregate(
col("input_column"),
map(),
(acc, c) => map_concat(acc, map(col("name"), col("other"))))))
This creates my_column with type map<string, strcut<...>>. Is there a way to make it struct<string, struct<...>>?
P.S. similar question - How to convert array of struct into struct of struct in Spark?
The following converts map to struct (map keys become struct fields).
val json_col = to_json($"col_map")
val json_schema = spark.read.json(df.select(json_col).as[String]).schema
val df2 = df.select(from_json(json_col, json_schema).alias("col_struct"))
In your case, it could look like this:
case class Strct(struct_name: String, struct_key: String)
val df = Seq(
Map("x" -> Strct("x", "val1"), "y" -> Strct("y", "val2"))
).toDF("map_of_structs")
df.printSchema()
// root
// |-- map_of_structs: map (nullable = true)
// | |-- key: string
// | |-- value: struct (valueContainsNull = true)
// | | |-- struct_name: string (nullable = true)
// | | |-- struct_key: string (nullable = true)
val json_col = to_json($"map_of_structs")
val json_schema = spark.read.json(df.select(json_col).as[String]).schema
val df2 = df.select(from_json(json_col, json_schema).alias("struct_of_structs"))
df2.printSchema()
// root
// |-- struct_of_structs: struct (nullable = true)
// | |-- x: struct (nullable = true)
// | | |-- struct_key: string (nullable = true)
// | | |-- struct_name: string (nullable = true)
// | |-- y: struct (nullable = true)
// | | |-- struct_key: string (nullable = true)
// | | |-- struct_name: string (nullable = true)
my_columnin order to create the new struct. You can't use struct field aliases from column values in Spark.cis not used