An error in groupby function in pyspark code

Question

I have a dataset on which I was asked to write a pyspark code for the following question.

List of Winners of Each World champions Trophy Hint: Total Result of all rounds of Tournament for that player is considered as that player's
Score/Result.
Result attributes: winner, tournament_name

I wrote this code:

game_info = spark.read.load("/content/chess/chess_wc_history_game_info.csv",
                     format="csv", sep=",", inferSchema="true", header="true")

game_info.groupBy('winner').show()

But on execution I got an error as:

AttributeError: 'GroupedData' object has no attribute 'show'

the error is quite explanatory. there is no show() method on df.groupBy(). the agg() has to accompany it. — samkart
– samkart, Commented Aug 9, 2022 at 10:17
what do you want to achieve ? what are you trying to compute ? — Steven
– Steven, Commented Aug 9, 2022 at 10:19
List of Winners of Each World champions Trophy Hint: Total Result of all rounds of Tournament for that player is considered as that player's Score/Result. Result attributes: winner, tournament_name this is what I have to find out — Samar Pratap Singh
– Samar Pratap Singh, Commented Aug 10, 2022 at 17:50

samkart · Accepted Answer · 2022-08-09 10:44:45Z

4

This error is there because groupBy() contains only below mentioned functions:

count() - Returns the count of rows for each group.
mean() - Returns the mean of values for each group.
max() - Returns the maximum of values for each group.
min() - Returns the minimum of values for each group.
sum() - Returns the total for values for each group.
avg() - Returns the average for values for each group.
agg() - Using agg() function, we can calculate more than one aggregation at a time.
pivot() - This function is used to Pivot the DataFrame.

edited Aug 9, 2022 at 10:44

samkart

6,7133 gold badges19 silver badges35 bronze badges

answered Aug 9, 2022 at 10:22

numb

631 silver badge5 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Samar Pratap Singh Over a year ago

so for this question do i need to apply max() function?

Guy Melul · Accepted Answer · 2022-08-09 13:53:38Z

1

I want to add another usefull function to @numb's list

collect_list - Collects all the values for a specific column foreach group

I guess this would help to "see" the groups

side note: truncate=False in show method print the table without truncating long text so you can actually see all the values

from pyspark.sql.functions import collect_list

game_info.groupBy('winner').agg(collect_list("<column you want to fetch>").alias('group_values')).show(truncate=False)

answered Aug 9, 2022 at 13:53

Guy Melul

1593 bronze badges

Collectives™ on Stack Overflow

An error in groupby function in pyspark code

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related