Bug: impossible to delete infinite values from DataFrame

Question

This is my DataFrame df:

col1       col2
-0.441406  2.523047
-0.321105  1.555589 
-0.412857  2.223047
-0.356610  2.513048

When I check df, I see that there are some infinite values.

np.any(np.isnan(df))
np.all(np.isfinite(df))

False 
True

What is the difference between NaN and infinite? Also, how can I delete all infinite values to get True in np.all(np.isfinite(X))?

This is what I tried:

df = df.replace([np.inf, -np.inf], np.nan).dropna(how="all")

But still the check of infinite gives me True.

Moreover, .apply(lambda s: s[np.isfinite(s)].dropna()).count() gives me the same number of rows of all columns as simply df.shape, which indicates the lack of infinite values. But in this case why np.all(np.isfinite(df)) returns True?

A. STEFANI · Accepted Answer · 2018-07-02 10:08:44Z

1

Your question is similar to dropping infinite values from dataframes in pandas?, did you try:

df.replace([np.inf, -np.inf], np.nan).dropna(subset=["col1", "col2"], how="all")

np.nan is not considered as finite, you may replace np.nan by any finite number see that code for example:

import pandas as pd
import numpy as np

df = pd.DataFrame(columns=list('ABC'))
df.loc[0] = [1,np.inf,-np.inf]
print df

print np.all(np.isfinite(df))

df_nan = df.replace([np.inf, -np.inf], np.nan).dropna(subset=df.columns, how="all")
print df_nan

print np.all(np.isfinite(df_nan))

df_0 = df.replace([np.inf, -np.inf], 0).dropna(subset=df.columns, how="all")
print df_0

print np.all(np.isfinite(df_0))

Result:

     A    B    C
0  1.0  inf -inf
False
     A   B   C
0  1.0 NaN NaN
False
     A    B    C
0  1.0  0.0  0.0
True

edited Jul 2, 2018 at 10:08

answered Jun 28, 2018 at 15:36

A. STEFANI

6,7711 gold badge25 silver badges49 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

ScalaBoy Over a year ago

How is it different from what I posted in my question? This is exactly what I tried and it didn't work.

A. STEFANI Over a year ago

Not exactly the same because : .dropna(subset=["col1", "col2"], how="all") != .dropna()

ScalaBoy Over a year ago

Should I mention all the columns? Can I do .dropna(subset=df.columns, how="all")?

ScalaBoy Over a year ago

I added the screenshot of my Jupyter Notebook to my post.

A. STEFANI Over a year ago

Yes you can do : subset=df.columns

Collectives™ on Stack Overflow

Bug: impossible to delete infinite values from DataFrame

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related