Dropping rows in Python using != operator is not working

Question

I want to drop rows in my dataset using:

totes = df3.loc[(df3['Reporting Date'] != '18/08/2017') & (df3['Business Line'] != 'Bondy')]

However it is not what I expect; I know that the number of rows I want ot drop is 496 after using:

totes = df3.loc[(df3['Reporting Date'] == '18/08/2017') & (df3['Business Line'] == 'Bondy')]

When I run my drop function, it is giving back much less rows than my dataset minus 496.

Does anyone know how to fix this?

Are u sure you should be using &. Maybe you meant "and" & is a bitwise and operator. — Hongyu Wang
– Hongyu Wang, Commented Aug 24, 2018 at 16:28
You can't just negate the components of a conjunction to negate the conjunction. (See de Morgan's laws.) — user2357112
– user2357112, Commented Aug 24, 2018 at 16:29
@HongyuWang: & is correct. and isn't overloadable, so libraries like NumPy and Pandas overload &. — user2357112
– user2357112, Commented Aug 24, 2018 at 16:30

jpp · Accepted Answer · 2018-08-24 16:45:26Z

2

You are correct to use &, but it is being misused. This is a logic problem. Note:

(NOT X) AND (NOT Y) != NOT(X AND Y)

Instead, you can calculate the negative of a Boolean condition via the ~ operator:

totes = df3.loc[~((df3['Reporting Date'] == '18/08/2017') & (df3['Business Line'] == 'Bondy'))]

Those parentheses and masks can get confusing, so you can write this more clearly:

m1 = df3['Reporting Date'].eq('18/08/2017')
m2 = df3['Business Line'].eq('Bondy')

totes = df3.loc[~(m1 & m2)]

Alternatively, note that:

NOT(X & Y) == NOT(X) | NOT(Y)

So you can use:

m1 = df3['Reporting Date'].ne('18/08/2017')
m2 = df3['Business Line'].ne('Bondy')

totes = df3.loc[m1 | m2]

edited Aug 24, 2018 at 16:45

answered Aug 24, 2018 at 16:33

jpp

166k37 gold badges301 silver badges363 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

ShadowRanger Over a year ago

Presumably a slightly more direct approach (with fewer parentheses) would be to use their original != query, just replace & with |, making it totes = df3.loc[(df3['Reporting Date'] != '18/08/2017') | (df3['Business Line'] != 'Bondy')]. That's just distributing the "not"/~, and it reads cleanly as "keep the stuff where the reporting date isn't the bad date or the business line isn't Bondy". I'm assuming pandas overloaded | to mean or the same way it overloaded & to mean and.

jpp Over a year ago

@ShadowRanger, Good point. Basically: NOT(X & Y) == NOT(X) | NOT(Y).

Collectives™ on Stack Overflow

Dropping rows in Python using != operator is not working

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related