5

I have a pandas dataframe, I want to check for each row if it has the same value at a particular column(let's call it porduct_type), and if it does, delete it. In other words, out of a group of consecutive rows with the same value at a particular column, I want to keep only one.

Example, if column A is the one on which we don't want consecutive duplicates:

input =  
A    B

    0  1    1
    0  2    2
    2  1   10
    2  2   20
    0  11  100
    5  2  200

output =  
A    B

    0  1    1
    2  1   10
    0  11  100
    5  2  200
Improve this question
1

1 Answer 1

Reset to default
5

It's a little tricky, but you could do something like

>>> df.groupby((df["A"] != df["A"].shift()).cumsum().values).first()
   A   B    C
1  0   1    1
2  2   1   10
3  0  11  100
4  5   2  200
Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

How about this df = df[df['A'] != df.shift(-1)['A']]
@BaronYugovich I would rather do df = df[df['A'] != df['A'].shift(-1)] - first ['A'] then shift(-1) to shift only one column not all df.
Does this solution only remove one consecutive duplicate? What if there are more than two consecutive rows with the same value in A?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.