Pandas Rows with missing values in multiple columns

Question

I have a dataframe with columns age, date and location.

I would like to count how many rows are empty across ALL columns (not some but all in the same time). I have the following code, each line works independently, but how do I say age AND date AND location isnull?
```
df['age'].isnull().sum()
df['date'].isnull().sum()
df['location'].isnull().sum()
```
I would like to return a dataframe after removing the rows with missing values in ALL these three columns, so something like the following lines but combined in one statement:
```
df.mask(row['location'].isnull())
df[np.isfinite(df['age'])]
df[np.isfinite(df['date'])]
```

Vaishali · Accepted Answer · 2019-01-30 21:36:37Z

1

You basically can use your approach, but drop the column indices:

df.isnull().sum().sum()

The first .sum() returns a per-column value, while the second .sum() will return the sum of all NaN values.

Similar to Vaishali's answer, you can use df.dropna() to drop all values that are NaN or None and only return your cleaned DataFrame.

In [45]: df = pd.DataFrame({'age': [1, 2, 3, np.NaN, 4, None], 'date': [1, 2, 3, 4, None, 5], 'location': ['a', 'b', 'c', None, 'e', 'f']})

In [46]: df
Out[46]: 
   age  date location
0  1.0   1.0        a
1  2.0   2.0        b
2  3.0   3.0        c
3  NaN   4.0     None
4  4.0   NaN        e
5  NaN   5.0        f

In [47]: df.isnull().sum().sum()
Out[47]: 4

In [48]: df.dropna()
Out[48]: 
   age  date location
0  1.0   1.0        a
1  2.0   2.0        b
2  3.0   3.0        c

edited Jan 30, 2019 at 21:36

Vaishali

38.5k5 gold badges62 silver badges88 bronze badges

answered May 2, 2017 at 20:02

Michael Gecht

1,4441 gold badge18 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Vaishali · Accepted Answer · 2017-05-02 20:05:12Z

1

You can find the no of rows with all NaNs by

len(df) - len(df.dropna(how = 'all'))

and drop by

df = df.dropna(how = 'all')

This will drop the rows with all the NaN values

edited May 2, 2017 at 20:05

answered May 2, 2017 at 19:53

Vaishali

38.5k5 gold badges62 silver badges88 bronze badges

Collectives™ on Stack Overflow

Pandas Rows with missing values in multiple columns

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related