Delete duplicate rows with the same value in all columns in pandas

Question

I have a dataframe with about a half a million rows. As I could see, there are plenty of duplicate rows, so how can I drop duplicate rows that have the same value in all of the columns (about 80 columns), not just one?

df:

period_start_time    id    val1    val2    val3
06.13.2017 22:00:00  i53    32      2       10
06.13.2017 22:00:00  i32    32      2       10
06.13.2017 22:00:00  i32    4       2       8
06.13.2017 22:00:00  i32    4       2       8
06.13.2017 22:00:00  i32    4       2       8
06.13.2017 22:00:00  i20    7       7       22
06.13.2017 22:00:00  i20    7       7       22

Desired output:

period_start_time    id    val1    val2    val3
06.13.2017 22:00:00  i53    32      2       10
06.13.2017 22:00:00  i32    32      2       10
06.13.2017 22:00:00  i32    4       2       8
06.13.2017 22:00:00  i20    7       7       22

did you try calling df.drop_duplicates()?

EdChum
– EdChum

2017-06-26 12:35:25 +00:00
Commented Jun 26, 2017 at 12:35 — EdChum
– EdChum, Commented Jun 26, 2017 at 12:35

jezrael · Accepted Answer · 2017-06-26 12:35:13Z

13

Use drop_duplicates:

df = df.drop_duplicates()
print (df)
     period_start_time   id  val1  val2  val3
0  06.13.2017 22:00:00  i53    32     2    10
1  06.13.2017 22:00:00  i32    32     2    10
2  06.13.2017 22:00:00  i32     4     2     8
5  06.13.2017 22:00:00  i20     7     7    22

answered Jun 26, 2017 at 12:35

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Delete duplicate rows with the same value in all columns in pandas

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related