map function on StructType in PySpark

Question

I have a StructType as follows:

to_Schema = StructType([StructField('name', StringType(), True),
           StructField('sales', IntegerType(), True)])

The dataframe_1 has both fields as StringType. So I created the above StructType so that I could use it to typecast the fields in dataframe_1.

I am able to do it in Scala:

val df2 = dataframe_1.selectExpr(to_Schema.map(
  col => s"CAST ( ${col.name} As ${col.dataType.sql}) ${col.name}"
): _*)

I am not able to use the same map function in python as StructType has no map function.

I've tried using for loop but it doesn't work as expected.

I am looking for a PySpark equivalent of the above Scala code.

viggnah · Accepted Answer · 2022-08-07 06:31:53Z

2

The below code will achieve the same thing in python:

for s in to_Schema:
    df = df.withColumn(s.name, df[s.name].cast(s.dataType))

You can also create a new dataframe from the old one using the new schema as shown in this answer:

df2 = spark.createDataFrame(dataframe_1.rdd, to_Schema)

answered Aug 7, 2022 at 6:31

viggnah

1,9171 gold badge6 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

ZygD · Accepted Answer · 2022-08-07 19:53:29Z

1

This would be the direct translation:

df2 = dataframe_1.selectExpr(*[f"CAST ({c.name} AS {c.dataType.simpleString()}) {c.name}" for c in to_Schema])

It could be simplified:

df2 = dataframe_1.select([col(c.name).cast(c.dataType).alias(c.name) for c in to_Schema])

However, I like this answer more ;)

answered Aug 7, 2022 at 19:53

ZygD

24.8k41 gold badges106 silver badges144 bronze badges