Replace Nulls of Struct type column using Pyspark

Question

I have a use case where I want to replace nulls in StructType columns with empty values. Below is sample which you can use to recreate the scenario:

from pyspark.sql.types import StructType, StructField, StringType, IntegerType

data = [(1, [1, "Aman"]), (2, [2, "Raman"]), (3, [3, "Baman"]), (4, None)]

schema = StructType(
    [
        StructField("ID", IntegerType(), True),
        StructField(
            "Name",
            StructType(
                [
                    StructField("NameID", IntegerType(), True),
                    StructField("FirstName", StringType(), True),
                ]
            ),
            True,
        ),
    ]
)

df = spark.createDataFrame(data, schema)

df.show()

I want to know how we can replace nulls in Name column.

blackbishop · Accepted Answer · 2022-01-31 11:51:10Z

2

Using when expression:

from pyspark.sql import functions as F

df1 = df.withColumn(
    "Name",
    F.when(
        F.col("Name").isNull(),
        F.struct(
            F.lit(None).alias("NameID"), #replace None by default ID value if needed
            F.lit("").alias("FirstName")
        )
    ).otherwise(F.col("Name"))
)

df1.show()
#+---+----------+
#| ID|      Name|
#+---+----------+
#|  1| {1, Aman}|
#|  2|{2, Raman}|
#|  3|{3, Baman}|
#|  4|  {null, }|
#+---+----------+

answered Jan 31, 2022 at 11:51

blackbishop

32.8k11 gold badges61 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Steven · Accepted Answer · 2022-01-31 13:08:20Z

1

Here is a solution with coalesce:

from pyspark.sql import functions as F

df1 = df.withColumn(
    "Name",
    F.coalesce(
        F.col("Name"),
        F.struct(
            F.lit(None).alias("NameID"),
            F.lit("").alias("FirstName"),
        ),
    ),
)

answered Jan 31, 2022 at 13:08

Steven

15.4k7 gold badges49 silver badges80 bronze badges

Collectives™ on Stack Overflow

Replace Nulls of Struct type column using Pyspark

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related