How to get max value group by another column from Pandas dataframe

Question

I have the following dataframe. I would like to get the rows where the date is max for each pipeline_name

Here is the dataframe:

+----+-----------------+--------------------------------------+----------------------------------+
|    | pipeline_name   | runid                                | run_end_dt                       |
|----+-----------------+--------------------------------------+----------------------------------|
|  0 | test_pipeline   | test_pipeline_run_101                | 2021-03-10 20:01:26.704265+00:00 |
|  1 | test_pipeline   | test_pipeline_run_102                | 2021-03-13 20:08:31.929038+00:00 |
|  2 | test_pipeline2  | test_pipeline2_run_101               | 2021-03-10 20:13:53.083525+00:00 |
|  3 | test_pipeline2  | test_pipeline2_run_102               | 2021-03-12 20:14:51.757058+00:00 |
|  4 | test_pipeline2  | test_pipeline2_run_103               | 2021-03-13 20:17:00.285573+00:00 |

Here is the result I want to achieve:

+----+-----------------+--------------------------------------+----------------------------------+
|    | pipeline_name   | runid                                | run_end_dt                       |
|----+-----------------+--------------------------------------+----------------------------------|
|  0 | test_pipeline   | test_pipeline_run_102                | 2021-03-13 20:08:31.929038+00:00 |
|  1 | test_pipeline2  | test_pipeline2_run_103               | 2021-03-13 20:17:00.285573+00:00 |

In the expected result, we have only the runid against each pipeline_name with the max run_end_dt

Thanks

Anurag Dabas · Accepted Answer · 2021-03-14 12:13:33Z

1

Suppose your dataframe stored in a variable named df

Just use groupby() method:-

df.groupby('pipeline_name',as_index=False)[['runid','run_end_dt']].max()

edited Mar 14, 2021 at 12:13

answered Mar 14, 2021 at 12:10

Anurag Dabas

24.3k9 gold badges25 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

soumith · Accepted Answer · 2021-03-14 12:11:25Z

0

Use groupby followed by a transform. Get the indices of the rows which have the max value in each group.

idx = (df.groupby(['pipeline_name'], sort=False)['run_end_dt'].transform('max') == df['run_end_dt'])
df = df.loc[idx]

answered Mar 14, 2021 at 12:11

soumith

6161 gold badge4 silver badges14 bronze badges

Collectives™ on Stack Overflow

How to get max value group by another column from Pandas dataframe

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related