Pyspark dataframe get all values of a column

Question

I want to get all values of a column in pyspark dataframe. I did some search, but I never find a efficient and short solution.

Assuming I want to get a values in the column called "name". I have a solution:

sum(dataframe.select("name").toPandas().values.tolist(),[])

It works, but it is not efficient since it converts to pandas then flatten the list... Is there a better and short solution?

There's no fast and efficient way to do it, but you can do way better than using sum to flatten a list of lists. The best you're going to get is probably: [x["name"] for x in dataframe.select("name").collect()] — pault
– pault, Commented Sep 5, 2019 at 17:11

SMaZ · Accepted Answer · 2019-09-06 01:15:59Z

31

Below Options will give better performance than sum.

Using collect_list

import pyspark.sql.functions as f
my_list = df.select(f.collect_list('name')).first()[0]

Using RDD:

my_list = df.select("name").rdd.flatMap(lambda x: x).collect()

I am not certain but in my couple of stress test, collect_list gives better performance. Will be great if someone can confirm.

answered Sep 6, 2019 at 0:34

SMaZ

2,6551 gold badge14 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Thanks. what is the f.collect_list('name')? I got error 'DataFrame' object has no attribute 'collect_list'

@chen you need to import pyspark functions with import pyspark.sql.functions as f

kartik · Accepted Answer · 2022-06-02 09:58:05Z

0

To get all values of a column in a list we can use collect() -

column_value_list = [row['<column_name>'] for row in df.select('<column_name>').collect()]

answered Jun 2, 2022 at 9:58

kartik

1171 silver badge3 bronze badges