How to add a element inside an element in a struct which is a element in struct in a spark Dataframe in spark scala

Question

I need to add an element inside an struct which is inside an struct itself.

File:

{"teamName":{"Redbull"},"info":{"drivers":{"driver":{"Max Verstappen","Alex Albon"},"carNumbers":{"33","23"},"carName":"RB7"}}}

base DF:

val jsonDF=spark.read.json("path")
jsonDF.printSchema

root
 |-- info: struct (nullable = true)
 |    |-- drivers: struct (nullable = true)
 |    |    |-- carName: string (nullable = true)
 |    |    |-- carNumbers: string (nullable = true)
 |    |    |-- driver: string (nullable = true)
 |-- teamName: string (nullable = true)

I need to add age inside,info -> drivers ->

When i do this

jsonDF.withColumn("info",struct(col("info.drivers").alias("drivers"), lit("24").alias("age"))).printSchema

root
 |-- info: struct (nullable = false)
 |    |-- drivers: struct (nullable = true)
 |    |    |-- carName: string (nullable = true)
 |    |    |-- carNumbers: string (nullable = true)
 |    |    |-- driver: string (nullable = true)
 |    |-- age: string (nullable = false)
 |-- teamName: string (nullable = true)

I'm getting it under info, i need to get it inside driver, how can i do that?

Saša Zejnilović · Accepted Answer · 2020-09-19 16:31:47Z

2

I would go with a library called spark-hats https://github.com/AbsaOSS/spark-hats

Then it is

import za.co.absa.spark.hats.Extensions._

val jsonDFwithAge = jsonDF.nestedWithColumn("info.drivers.age", lit("24"))

jsonDFwithAge.printSchema
root
 |-- info: struct (nullable = false)
 |    |-- drivers: struct (nullable = false)
 |    |    |-- carName: string (nullable = true)
 |    |    |-- carNumbers: string (nullable = true)
 |    |    |-- driver: string (nullable = true)
 |    |    |-- ag: string (nullable = false)
 |-- teamName: string (nullable = true)

answered Sep 19, 2020 at 16:31

Saša Zejnilović

9207 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to add a element inside an element in a struct which is a element in struct in a spark Dataframe in spark scala

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related