Using pyspark groupBy with a custom function in agg

Question

I would like to groupBy my spark df with custom agg function:

def gini(list_of_values):
    sth is processing here
    return number output

enter image description here

I would like to get sth like that:

df.groupby('activity')['mean_event_duration_in_hours].agg(gini)

Could you please help me to resolve this tackle?

Jan Jaap Meijerink · Accepted Answer · 2022-01-20 14:04:42Z

3

You can create a udf like so:

import pyspark.sql.functions as F
from pyspark.sql.types import FloatType

def gini(list_of_values):
    # sth is processing here
    return number_output

udf_gini = F.udf(gini, FloatType())

df.groupby('activity')\
    .agg(F.collect_list("mean_event_duration_in_hours").alias("event_duration_list"))\
    .withColumn("gini", udf_gini(F.col("event_duration_list")))

Or define gini as a UDF like this:

@udf(returnType=FloatType())
def gini(list_of_values):
    # sth is processing here
    return number_output

df.groupby('activity')\
    .agg(F.collect_list("mean_event_duration_in_hours").alias("event_duration_list"))\
    .withColumn("gini", gini(F.col("event_duration_list")))

edited Jan 20, 2022 at 14:04

answered Jan 20, 2022 at 12:48

Jan Jaap Meijerink

4472 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Using pyspark groupBy with a custom function in agg

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related