How to add the index of the array as a field to an array of structs in pyspark dataframe

Question

I have a dataframe containing an array of structs. I would like to add the index of the array as a field within the struct. Is this possible?

So structure would go from:

|-- my_array_column: array
 |    |-- element: struct
 |    |    |-- field1: string
 |    |    |-- field2: string

to:

|-- my_array_column: array
 |    |-- element: struct
 |    |    |-- field1: string
 |    |    |-- field2: string
 |    |    |-- index of element: integer

Many thanks

blackbishop · Accepted Answer · 2022-07-29 09:56:14Z

4

For Spark 3.1+, you can use transform function and withField to update each struct element of the array column like his:

from pyspark.sql import functions as F

df = df.withColumn(
    "my_array_column",
    F.transform("my_array_column", lambda x, i: x.withField("index", i))
)

For older version, you'll have to recreate the whole struct element in order to add a field:

df = df.withColumn(
    "my_array_column",
    F.expr("transform(my_array_column, (x, i) -> struct(x.field1 as field1, x.field2 as field2, i as index))")
)

answered Jul 29, 2022 at 9:56

blackbishop

32.8k11 gold badges61 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

samkart Over a year ago

noice! much better and straightforward than using aggregate()

steveWB Over a year ago

This is perfect, i didn't realise transform could access the index of the element in the second argument of the lambda like that. Documentation for reference: spark.apache.org/docs/3.2.0/api/python/reference/api/…

Collectives™ on Stack Overflow

How to add the index of the array as a field to an array of structs in pyspark dataframe

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related