How to add an array of list as a new column to a spark dataframe using pyspark

Question

I have an array of list.

array([1,2,3,4.....,50])

I want to add each element in this list as a new column to my current spark dataframe. Something like this

col1|col2|new_col
-------------------
a   |aa  |1
-------------------
b   |bb  |2
-------------------
c   |cc  |3
-------------------
d   |dd  |4
-------------------
...
...
...
...
-------------------
z  |zz  |50
--------------------

withColumn seems to not work for something like this.

s.polam · Accepted Answer · 2020-11-19 23:06:01Z

3

Use arrays_zip function, for this first we need to convert existing data into array & then use arrays_zip function to combine existing and new list of data. Check below code.

list_data = [1,2,3,4.....,50]

df \
.select(collect_list(struct(F.col("*"))).alias("data")) \
.withColumn("list",F.array([F.lit(i) for i in list_data])) \
.select(F.explode(F.arrays_zip(F.col("data"),F.col("list"))).alias("full_data")) \ 
.select(F.col("full_data.data.*"),F.col("full_data.list").alias("col3")) \ 
.show()

Spark Version >= 2.4.0

edited Nov 19, 2020 at 23:06

answered Nov 19, 2020 at 20:46

s.polam

10.4k2 gold badges17 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

smn Over a year ago

getting an error 'numpy.int64' object has no attribute '_get_object_id'

s.polam Over a year ago

Spark version??

smn Over a year ago

Spark 3.0.1 Right at the withColumn statement. Array is somehow causing the problem. Used an ordinary list and it's working but array of list isn't.

s.polam Over a year ago

Ok..I have copied that from your question and pasted .. now updated solution check once

smn Over a year ago

Yea I know list is working. I don't want a list. I have an array of list. I want that as a new column. Even if I convert my array of list into a list and then use that list, It's giving out the same error. I tried with a sample list like [1,2,3], then it's working. Even if I am changing my array([list]) into a list using list(array) and then use that list. It's giving out the same error.

|

Collectives™ on Stack Overflow

How to add an array of list as a new column to a spark dataframe using pyspark

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related