pandas `to_numeric` integer downcast cast floats not to integer

Question

With this sample dataframe:

>>> d = pd.DataFrame({'si': ['1', '2', 'NA'], 's': ['a', 'b', 'c']})

>>> d.dtypes
#
si    object
s     object
dtype: object

My first attempt was to use astype and the 'Int64' NA aware int type, but I got a

traceback

>>> d.si.astype('Int64')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-144-ed289e0c95aa> in <module>
----> 1 d.si.astype('Int64')
...

then I try the `to_numeric` method:

pandas to_numeric integer downcast cast floats

In [112]: d.loc[:, 'ii'] = pd.to_numeric(d.si, errors='coerce', downcast='integer')

In [113]: d.dtypes
Out[113]: 
si     object
s      object
ii    float64
dtype: object

In [114]: d
Out[114]: 
    si  s   ii
0    1  a  1.0
1    2  b  2.0
2   NA  c  NA

In the above I expect to have ii column with integers and integer nan

Documentation say:

downcast : {'integer', 'signed', 'unsigned', 'float'}, default None
    If not None, and if the data has been successfully cast to a
    numerical dtype (or if the data was numeric to begin with),
    downcast that resulting data to the smallest numerical dtype
    possible according to the following rules:

    - 'integer' or 'signed': smallest signed int dtype (min.: np.int8)
    - 'unsigned': smallest unsigned int dtype (min.: np.uint8)
    - 'float': smallest float dtype (min.: np.float32)

The key word is possible. With errors='coerce', you expect things to get set to NaN, which is a float, so that's the smallest type possible. If you want you can .astype('Int64') to make it the nullable integer type — ALollz
– ALollz, Commented Jun 6, 2022 at 16:55
Thank you @Alollz, I've updated my question to illustrate that .astype on this simple dataframe traceback — user3313834
– user3313834, Commented Jun 6, 2022 at 17:32

user3313834 · Accepted Answer · 2022-06-06 18:00:10Z

4

Unfortunately, pandas is still adapting/transitioning to fully supporting integer NaN. For that, you have to explicitly convert it to Int64 after your pd.to_numeric operation.

No need to downcast.

# Can also use `'Int64' as dtype below.
>>> pd.to_numeric(df['col'], errors='coerce').astype(pd.Int64Dtype())

# or

>>> pd.to_numeric(df['col'], errors='coerce').astype('Int64')

0       1
1       2
2       3
3    <NA>
Name: col, dtype: Int64

edited Jun 6, 2022 at 18:00

user3313834

7,91718 gold badges65 silver badges112 bronze badges

answered Jun 6, 2022 at 17:01

rafaelc

59.4k15 gold badges64 silver badges87 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

user17242583 · Accepted Answer · 2022-06-06 16:56:46Z

2

You have errors='coerce' set, and the documentation for that option says (emphasis mine):

errors : {'ignore', 'raise', 'coerce'}, default 'raise'

If 'raise', then invalid parsing will raise an exception.

If 'coerce', then invalid parsing will be set as NaN.

If 'ignore', then invalid parsing will return the input.

Since your si column contains NaNs, you can't convert it to an integer column because NaN is a float, and therefore all other values in the column are upcasted to the float64 dtype.

answered Jun 6, 2022 at 16:56

user17242583

Collectives™ on Stack Overflow

pandas `to_numeric` integer downcast cast floats not to integer

My first attempt was to use astype and the 'Int64' NA aware int type, but I got a

then I try the `to_numeric` method:

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

My first attempt was to use astype and the 'Int64' NA aware int type, but I got a

then I try the to_numeric method:

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related

then I try the `to_numeric` method: