How to count frequency of elements from a columns of lists in pyspark dataframe?

Question

I have a pyspark dataframe that looks like the following,

data2 = [("James",["A x","B z","C q","D", "E"]),
    ("Michael",["A x","C","E","K", "D"]),
    ("Robert",["A y","R","B z","B","D"]),
    ("Maria",["X","A y","B z","F","B"]),
    ("Jen",["A","B","C q","F","R"])
  ]

 
df2 = spark.createDataFrame(data2, ["Name", "My_list" ])

df2
    Name    My_list
0   James   [A x, B z, C q, D, E]
1   Michael     [A x, C, E, K, D]
2   Robert  [A y, R, B z, B, D]
3   Maria   [X, A y, B z, F, B]
4   Jen     [A, B, C q, F, R]

I want to be able to count the elements in the column 'My_list' and sort in descending order? For example,

'A x' appeared -> P times, 
'B z' appeared -> Q times, and so on.

Can someone please put some lights on this? Thank you very much in advance.

Anna K. · Accepted Answer · 2021-11-16 01:27:56Z

5

The following command explodes the array, and provides the count of each element

import pyspark.sql.functions as F

df_ans = (df2
           .withColumn("explode", F.explode("My_list"))
           .groupBy("explode")
           .count()
           .orderBy(F.desc("count"))

the result is

answered Nov 16, 2021 at 1:27

Anna K.

1,57214 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to count frequency of elements from a columns of lists in pyspark dataframe?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related