float type appear when use replace method in pandas dataframe

Question

import pandas as pd
import numpy as np


df = pd.DataFrame(
    [
        [np.nan, 'None', 3],
        [np.nan, 5, 6],
        [7, 8, 9]
     ], columns=['a', 'b', 'c']
)

df.replace({np.nan: None}, inplace=True)
print(df)

df.replace({'None': None}, inplace=True)
print(df)

      a     b  c
0  None  None  3
1  None     5  6
2     7     8  9
     a    b  c
0  NaN  NaN  3
1  NaN  5.0  6
2  7.0  8.0  9

this is small example fo my case. i wanna replace nan, "None" to None. so i use replace twice first replace method work fine as i thought, but nan was reborn in second replace and all int is changed to float because of nan. i have no idea about why nan is reborn df.replace({'None': None}, inplace=True), how can i fix it?

you need nullable int dtype: df.replace({'None': None}).astype(pd.Int64Dtype()), docs — anky
– anky, Commented Mar 3, 2021 at 2:38
Could I ask why you want to use None instead of NaN? For most cases, NaN is the best way to indicate a missing value in a DataFrame. — Ali
– Ali, Commented Mar 3, 2021 at 2:46

Ali · Accepted Answer · 2021-03-03 15:59:04Z

1

If you want integers in a column with nan values you need to use pd.NA instead. nan is a float and will force an array of integers to become a floating point. Check out the documentation.

Solution

df = pd.DataFrame(
    [
        [np.nan, None, 3],
        [np.nan, 5, 6],
        [7, 8, 9]
    ], 
    columns=['a', 'b', 'c'],
)

# replace np.nan with pd.NA
# then convert columns types to Int32
df.fillna(pd.NA).astype('Int32')

Out[11]:
      a     b  c
0  <NA>  <NA>  3
1  <NA>     5  6
2     7     8  9

edited Mar 3, 2021 at 15:59

answered Mar 3, 2021 at 3:20

Ali

3383 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

BENY · Accepted Answer · 2021-03-03 02:39:42Z

0

We can do is change it to object

out = df.replace({'None': None}).astype(object)
Out[10]: 
     a    b  c
0  NaN  NaN  3
1  NaN    5  6
2    7    8  9

answered Mar 3, 2021 at 2:39

BENY

324k22 gold badges176 silver badges250 bronze badges

1 Comment

PSW Over a year ago

even i dont use object type, your reply teach me variable way. thanks!

jeffhale · Accepted Answer · 2021-03-03 02:45:57Z

0

It sounds like you want the column type to be an integer instead of a float. You can use the nullable integer dtype introduced in pandas version 0.24.0.

With a column that is a regular integer dtype is automatically converted to a float dtype if it gets a null value. Note that if you use the pandas nullable integer dtype, the column will not become a float and the null value will be represented as <NA> the pandas.NA value.

1 Comment

PSW Over a year ago

i know nullable integer type through your reply. thanks!

Collectives™ on Stack Overflow

float type appear when use replace method in pandas dataframe

3 Answers 3

Comments

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related