Pandas dataframe selection based on content of list object in dataframe

Question

I have a pandas dataframe where one column contains list objects which gives me dtype: object. The list objects are of different lengths.

        features           other_features
0        ["foo", "bar"]                2
1        ["foo", "dog"]                1
2        ["dog"]                       4

I want to select the rows in my dataframe where the list has elements that are in a different list external_list = ["dog", "cat"].

In this case I would like the result to be the rows that contains dog or cat together with the other coulmns, which would be:

        features           other_features
1        ["foo", "dog"]                1
2        ["dog"]                       4

I have tried isin, but from what I understand it requires the column type of the element to look at to not be a list of more than one object.

filter = df["features"].isin(["dog", "cat"])

It results in every element being False, which it should not since some should be True, and it is not returning the rest of the columns.

How can I solve this?

jezrael · Accepted Answer · 2020-04-14 13:44:09Z

2

Use map with comparing list converted to sets tested by isdisjoint, ~ is for inverse mask:

#if string repr of list
#import ast
#df['features'] = df['features'].apply(ast.literal_eval)

external_list = ["dog", "cat"]
df = df[~df.features.map(set(external_list).isdisjoint)]
print (df)
     features  other_features
1  [foo, dog]               1
2       [dog]               4

edited Apr 14, 2020 at 13:44

answered Apr 14, 2020 at 13:36

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Becczor Over a year ago

Thank you! At first the resulting data frame was empty, the comments with literal_eval solved it and it gives me exactly what I wanted.

BENY · Accepted Answer · 2020-04-14 13:38:51Z

0

Let us do "explode" on axis =0, then we can do isin

m=pd.DataFrame(df.features.tolist(),index=df.index).isin(external_list).any(1)
df=df[~m]

answered Apr 14, 2020 at 13:38

BENY

324k22 gold badges176 silver badges250 bronze badges

Collectives™ on Stack Overflow

Pandas dataframe selection based on content of list object in dataframe

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related