update field in struct column pyspark - withField does not replace

Question

I have a table with a column with dates, which I want to use to update the value of field in a struct that I define for a new column.

Simple table:

date 
----------
1960-12-01

Struct:

value_type = T.StructType(
    [
       T.StructField("extra",T.MapType(T.StringType(), T.StringType(), True), True),
       T.StructField("date", T.StringType(), True),
       T.StructField("from_date", T.StringType(), True),
       T.StructField("to_date", T.StringType(), True),
       T.StructField("value", T.StringType(), True),
    ]
)

Which I use to create a new empty column which I then want to update with this code:

dateOfBirth_df = (
  df
 .withColumn("dob_new", (F.lit(None).cast(value_type)))
 .withColumn("dob_new", F.col("dob_new").withField("value", F.lit("date"))))

Which works, but it the dob_new.date field does not get updated. Anyone knows what I am doing wrong?

blackbishop · Accepted Answer · 2022-02-10 13:35:43Z

4

When you do:

F.lit(None).cast(value_type)

It is creating a column with value NULL. So withField on null value will always give null.

You can try this instead:

from pyspark.sql import functions as F

dateOfBirth_df = df.withColumn(
    "dob_new",
    F.struct(*[F.lit(None).cast(f.dataType).alias(f.name) for f in value_type.fields])
).withColumn(
    "dob_new",
    F.col("dob_new").withField("value", F.col("date"))
)

dateOfBirth_df.show(truncate=False)
#+----------+------------------------------------+
#|date      |dob_new                             |
#+----------+------------------------------------+
#|1960-12-01|{null, null, null, null, 1960-12-01}|
#+----------+------------------------------------+

answered Feb 10, 2022 at 13:35

blackbishop

32.8k11 gold badges61 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

update field in struct column pyspark - withField does not replace

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related