Pandas - Replace missing values & simultaneously add prefix or suffix based on column?

Question

I'm trying to pre-process some data for machine learning purposes. I'm currently trying to clean up some NaN values and replace them with 'unknown' and a prefix or suffix which is based on the column name.

The reason for this is when I'm use one hot encoding, I can't have multiple columns with the same name being fed into xgboost.

So what I have is the following

df = df.apply(lambda x: x.replace(np.nan, 'unknown'))

And I'd like to replace all instances of NaN in the df with 'unknown_columname'. Is there any easy or simple way to do this?

Try df = df.apply(lambda x: x.replace(np.nan, f'unknown_{x.name}')). You can also use df = df.apply(lambda x: x.fillna(f'unknown_{x.name}')) — goku
– goku, Commented Sep 9, 2020 at 21:57
This is perfect and goes along well with my code! If you submit this as an answer, I'd like to give you the points! — Grygger
– Grygger, Commented Sep 9, 2020 at 22:13

goku · Accepted Answer · 2020-09-09 22:16:35Z

2

Try df = df.apply(lambda x: x.replace(np.nan, f'unknown_{x.name}')).

You can also use df = df.apply(lambda x: x.fillna(f'unknown_{x.name}').

answered Sep 9, 2020 at 22:16

goku

1562 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

HadarM · Accepted Answer · 2020-09-09 22:02:40Z

1

First let's create the backup array to be filled whenever we have a missing value

s = np.core.defchararray.add('unknown',df.columns.values)

Then we can simply replace each NaN with the right value from s:

cols = df.columns.values
for col_name in cols:
    df.col_name.fillna(s, inplace=True)

answered Sep 9, 2020 at 22:02

HadarM

1131 silver badge9 bronze badges

Collectives™ on Stack Overflow

Pandas - Replace missing values & simultaneously add prefix or suffix based on column?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related