Add new column with literal value to a struct column in Dataframe in Spark Scala

Question

I have a dataframe with the following schema:

root
 |-- docnumber: string (nullable = true)
 |-- event: struct (nullable = false)
 |    |-- data: struct (nullable = true)
           |-- codevent: int (nullable = true)

I need to add a column inside event.data so that the schema would be like:

root
 |-- docnumber: string (nullable = true)
 |-- event: struct (nullable = false)
 |    |-- data: struct (nullable = true)
           |-- codevent: int (nullable = true)
           |-- needtoaddit: int (nullable = true)

I tried

dataframe.withColumn("event.data.needtoaddit", lit("added"))

but it adds a column with name event.data.needtoaddit

dataframe.withColumn(
  "event",
  struct(
    $"event.*",
    struct(
      lit("added")
        .as("needtoaddit")
    ).as("data")
  )
)

but it creates an ambiguous column named event.data and again I have a problem.

How can I make it work?

ZygD · Accepted Answer · 2022-07-18 06:48:12Z

3

Spark 3.1+

To add fields inside struct columns, use withField

col("event.data").withField("needtoaddit", lit("added"))

Input:

val df = spark.createDataFrame(Seq(("1", 2)))
    .select(
        col("_1").as("docnumber"),
        struct(struct(col("_2").as("codevent")).as("data")).as("event")
    )
df.printSchema()
// root
//  |-- docnumber: string (nullable = true)
//  |-- event: struct (nullable = false)
//  |    |-- data: struct (nullable = false)
//  |    |    |-- codevent: long (nullable = true)

Script:

val df2 = df.withColumn(
    "event",
    col("event.data").withField("needtoaddit", lit("added"))
)

df2.printSchema()
// root
//  |-- docnumber: string (nullable = true)
//  |-- event: struct (nullable = false)
//  |    |-- data: struct (nullable = true)
//            |-- codevent: int (nullable = true)
//            |-- needtoaddit: int (nullable = true)

edited Jul 18, 2022 at 6:48

answered Jul 18, 2022 at 6:39

ZygD

24.8k41 gold badges107 silver badges144 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

mck · Accepted Answer · 2021-04-07 13:37:45Z

1

You're kind of close. Try this code:

val df2 = df.withColumn(
    "event", 
    struct(
        struct(
            $"event.data.*", 
            lit("added").as("needtoaddit")
        ).as("data")
    )
)

answered Apr 7, 2021 at 13:37

mck

42.7k13 gold badges44 silver badges62 bronze badges

3 Comments

Anderson Dutra Over a year ago

after teste better it won't work, the event data was losted when i run this code

Anderson Dutra Over a year ago

if i add "event", struct($"event.*", struct($"event.data.*",... it won't work only add one column to data

mck Over a year ago

you should not add event. that will overwrite the new data column with the existing data column.

Collectives™ on Stack Overflow

Add new column with literal value to a struct column in Dataframe in Spark Scala

2 Answers 2

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related