Remove NaN from lists in Pandas dataframe [duplicate]

Question

I have a dataframe whose values are lists which contains 'nan'. Is there an easy and pythonic way to remove those 'nan' values from lists within the dataframe? I have defined a function which returns a list without 'nan', but how can I apply it to dataframe inplace?

def remove_nan(input_list):
    temp_list = [x for x in input_list if x!='nan']
    return temp_list

test = ['nan', 'nan', 'SHM System', 'nan', 'nan', 'nan']

remove_nan(test)
['SHM System']

This function works on individual list and returns clean list as shown in the output above. How can I apply this function, or if there is a better way, to remove all 'nan' values from lists within a dataframe? I have tried applymap and apply but didn`t work.

df_combined.applymap(remove_nan)

Show your bit of code that didn't work too... How to Ask minimal reproducible example — Julien
– Julien, Commented Mar 17, 2021 at 4:32
This is what I did: df_combined.applymap(remove_nan) It didn`t change the dataframe. — Gibrail Hassan
– Gibrail Hassan, Commented Mar 17, 2021 at 4:35
Did you try df_combined = df_combined.applymap(remove_nan) ? The applymap method doesn't modify the original DataFrame unless you set it equal to what it's returning — Derek O
– Derek O, Commented Mar 17, 2021 at 4:37
@DerekO Yeah, I can see the copy of this data frame, it is same. Hasn`t changed. — Gibrail Hassan
– Gibrail Hassan, Commented Mar 17, 2021 at 4:40

Gibrail Hassan · Accepted Answer · 2021-03-17 04:45:42Z

2

The following line of code worked for me. Thanks to @piRSquared.

df.applymap(lambda x: [*filter(pd.notna, x)])

answered Mar 17, 2021 at 4:45

Gibrail Hassan

858 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Remove NaN from lists in Pandas dataframe [duplicate]

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related