pandas dataframe groupby columns and aggregate on custom function

Question

I am trying to group a dataframe by certain columns and then for each group, pass its column series as a list to a custom function or lambda and get a single aggregated result.

Here's a df:

orgid.      appid.  p.  type.   version
-------------------------------------------------
24e78b      4ef36d  1   None    3.3.7
24e78b      4ef36d  2   None    3.4.1
24e78b      4ef36d  1   None    3.3.7-beta-1
24e78b      4ef36d  1   None    3.4.0-mvn.1
24e78b      4ef36d  2   None    3.4.0-beta.5
24e78b      4ef36d  1   None    3.4.0-beta.1
24e78b      4ef36d  1   None    3.4.0
24e78b      4ef36d  1   None    3.3.5

So I have a function that takes a list of versions and returns a max version string.

>> versions = ['3.4.0-mvn.1', '3.4.0-beta.1', '3.4.0', '3.3.7-beta-1', '3.3.7', '3.3.5', '3.4.0-beta-1']
>> str(max(map(semver.VersionInfo.parse, versions)))
'3.4.0'

Now I want to group the dataframe and then each group's version series is passed to this function as a list and return a single version string.

I tried:

>> g = df.groupby(['orgid', 'appid', 'p', 'type'])
>> g['version'].apply(lambda x: str(max(map(semver.VersionInfo.parse, x.tolist()))))
Series([], Name: version, dtype: float64)

I get a empty series.

Expected output:

orgid.      appid.  p.  type.   version
24e78b      4ef36d  1   None    3.4.0
24e78b      4ef36d  2   None    3.4.1

I am also referencing this Pandas group by multiple custom aggregate function on multiple columns post here.

But couldn't get it right.

can you give reproducible df?

s510
– s510

2022-09-12 20:33:16 +00:00
Commented Sep 12, 2022 at 20:33 — s510
– s510, Commented Sep 12, 2022 at 20:33

Andrej Kesely · Accepted Answer · 2022-09-12 20:04:41Z

2

Try:

import semver

df["version"] = df["version"].apply(semver.VersionInfo.parse)
out = df.groupby(["orgid", "appid", "p", "type"], as_index=False).max()

print(out)

Prints:

    orgid   appid  p  type version
0  24e78b  4ef36d  1  None   3.4.0
1  24e78b  4ef36d  2  None   3.4.1

answered Sep 12, 2022 at 20:04

Andrej Kesely

196k15 gold badges60 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Azima Over a year ago

this still prints me the Empty dataframe. :(

Alessandro Gianfelici · Accepted Answer · 2022-09-12 19:12:37Z

1

This happens because of the None values in the column you are passing to the groupby method.

Try to do:

df = df.fillna('None')

Before calling df.groupby(...), it should work.

answered Sep 12, 2022 at 19:12

Alessandro Gianfelici

443 bronze badges

Comments

BeRT2me · Accepted Answer · 2022-09-12 21:46:57Z

1

out = (df.groupby(['orgid', 'appid', 'p', 'type'], as_index=False)['version']
         .agg(lambda x: max(semver.VersionInfo.parse(v) for v in x)))
print(out)

# Output:

    orgid   appid  p  type version
0  24e78b  4ef36d  1  None   3.4.0
1  24e78b  4ef36d  2  None   3.4.1

answered Sep 12, 2022 at 21:46

BeRT2me

13.3k2 gold badges18 silver badges39 bronze badges

1 Comment

Azima Over a year ago

but this still prints me the Empty dataframe.

Collectives™ on Stack Overflow

pandas dataframe groupby columns and aggregate on custom function

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related