pandas.series.drop_duplicates() cannot delete all duplicates

Question

>>> info
0                       (dataset, license, sources, weight)
1                       (dataset, license, sources, weight)
2                       (dataset, license, sources, weight)
3                       (dataset, license, sources, weight)
4                       (dataset, license, sources, weight)
                                ...                        
491877    (dataset, license, sources, surfaceEnd, surfac...
491878    (dataset, license, sources, surfaceEnd, surfac...
491879    (dataset, license, sources, surfaceEnd, surfac...
491880    (dataset, license, sources, surfaceEnd, surfac...
491881    (dataset, license, sources, surfaceEnd, surfac...
Name: edge_info, Length: 491882, dtype: object

>>> info.drop_duplicates()
0                   (dataset, license, sources, weight)
1                   (dataset, license, sources, weight)
70    (dataset, license, sources, surfaceEnd, surfac...
71    (dataset, license, sources, surfaceEnd, surfac...
Name: edge_info, dtype: object

>>> info.iloc[0]==info.iloc[1]
True
>>> info.iloc[0]==info.iloc[2]
True
>>> info.iloc[0]
dict_keys(['dataset', 'license', 'sources', 'weight'])
>>>

The above commands require a series object to drop duplicate items.

However, the results seem that still have duplicate value as show above.

The first row info.iloc[0] and the second row info.iloc[1] of info is equal, but the drop_duplicates() function do not remove the second item.

Does any know the reason for the results?

What is print (info.drop_duplicates().head(2).tolist()) ? Because if use info.iloc[0]==info.iloc[1] is tested original info, not output from removed duplicates. Need assign back like info = info.drop_duplicates() and then test info.iloc[0]==info.iloc[1] — jezrael
– jezrael, Commented Jan 11, 2021 at 9:57

Patrick Potocny · Accepted Answer · 2021-01-11 15:58:21Z

1

The reason the first and second row is equal that you are still checking the original dataframe, not the output you removed duplicates from. Most functions (but not all, always check the documentation) in pandas that modify DataFrame/Series, they don't change the original data by default instead they just return the modified data.

To resolve your issue you have two options:

Assign the modified data to the variable::
info = info.drop_duplicates()
Set parameter inplace=True(by default this is False).This will change the dataframe object in place and will return None.

info.drop_duplicates(inplace=True)

Here are some useful links about both methods:

edited Jan 11, 2021 at 15:58

answered Jan 11, 2021 at 11:35

Patrick Potocny

1541 silver badge9 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

pandas.series.drop_duplicates() cannot delete all duplicates

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related