Add list as column to Dataframe in pyspark

Question

I have a list of integers and a sqlcontext dataframe with the number of rows equal to the length of the list. I want to add the list as a column to this dataframe maintaining the order. I feel like this should be really simple but I can't find an elegant solution.

Possible duplicate of PySpark: Add a column to DataFrame when column is a list — Rudr
– Rudr, Commented Jan 4, 2019 at 20:19

Mariusz · Accepted Answer · 2016-11-03 09:37:10Z

1

You cannot simply add a list as a dataframe column since list is local object and dataframe is distirbuted. You can try one of thw followin approaches:

convert dataframe to local by collect() or toLocalIterator() and for each row add corresponding value from the list OR
convert list to dataframe adding an extra column (with keys from dataframe) and then join them both

answered Nov 3, 2016 at 9:37

Mariusz

14k3 gold badges66 silver badges66 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Timothy Elbert Over a year ago

I ended up doing the second because collect or toLocalIterator would have overwhelmed the memory. The trouble was that it took me a while to figure out how to do the second point, which is partly why I asked the question. I didn't ask this explicitly because I was hoping there was a more elegant way.

Collectives™ on Stack Overflow

Add list as column to Dataframe in pyspark

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related