How to interact with each element of an ArrayType column in pyspark?

Question

If I have an ArrayType column in pyspark

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame(((1,[]),(2,[1,2,3]),(3,[-2])),schema=StructType([StructField("a",IntegerType()),StructField("b",ArrayType(IntegerType()))]))
df.show()
output:
+---+---------+
|  a|        b|
+---+---------+
|  1|       []|
|  2|[1, 2, 3]|
|  3|     [-2]|

Now, I want to be able to interact with each element of column b, Like,

Divide each element by 5 output:

+---+---------------+
|  a|              b|
+---+---------------+
|  1|             []|
|  2|[0.2, 0.4, 0.6]|
|  3|         [-0.4]|
+---+---------------+

Add to each element etc.

How do I go about such transformations where some operator or function is applied to each element of the array type columns?

Nithish · Accepted Answer · 2022-01-16 10:33:50Z

3

You are looking for the tranform function. Transform enables to apply computation on each element of an array.

from pyspark.sql import functions as F

# Spark < 3.1.0
df.withColumn("b", F.expr("transform(b, x ->  x / 5)")).show()

"""
+---+---------------+
|  a|              b|
+---+---------------+
|  1|             []|
|  2|[0.2, 0.4, 0.6]|
|  3|         [-0.4]|
+---+---------------+
"""

# Spark >= 3.1.0

df.withColumn("b", F.transform("b", lambda x: x / 5)).show()
"""
+---+---------------+
|  a|              b|
+---+---------------+
|  1|             []|
|  2|[0.2, 0.4, 0.6]|
|  3|         [-0.4]|
+---+---------------+
"""

answered Jan 16, 2022 at 10:33

Nithish

3,2472 gold badges11 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

figs_and_nuts Over a year ago

Ahh yess!! The documentation says : "Returns an array of elements after applying a transformation to each element in the input array.". I think they should have documented this under the array section in the documentation. And the name should not be so generic that gives no hint to google that it can show this under search results. Thank you for quick help and your time @Nithish

der_grund Over a year ago

I've searched for hours to find this! Thank you! One small addition for possible future readers. If you import pyspark.sql.functions as f, you must not use f.fun() within the transform expression. Just use the bare function name.

Collectives™ on Stack Overflow

How to interact with each element of an ArrayType column in pyspark?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related