Empty DataFrame doesn't admit its empty

Question

I must not understand something about emptiness when it comes to pandas DataFrames. I have a DF with empty rows but when I isolate one of these rows its not empty.

Here I've made a dataframe:

>>> df = pandas.DataFrame(columns=[1,2,3], data=[[1,2,3],[1,None,3],[None, None, None],[3,2,1],[4,5,6],[None,None,None],[None,None,None]])
>>> df
     1    2    3
0  1.0  2.0  3.0
1  1.0  NaN  3.0
2  NaN  NaN  NaN
3  3.0  2.0  1.0
4  4.0  5.0  6.0
5  NaN  NaN  NaN
6  NaN  NaN  NaN

Then I know row '2' is full of nothing so I check for that...

>>> df[2:3].empty
    False

Odd. So I split it out into its own dataframe:

>>> df1 = df[2:3]
>>> df1
    1   2   3
2 NaN NaN NaN

>>> df1.empty
False

How do I check for emptiness (all the elements in a row being None or NaN?)

http://pandas.pydata.org/pandas-docs/version/0.18/generated/pandas.DataFrame.empty.html

pandas.pydata.org/pandas-docs/version/0.18/generated/… check the website example , they already provide the solution — BENY
– BENY, Commented Nov 16, 2017 at 16:33

cs95 · Accepted Answer · 2017-11-16 16:19:16Z

2

You're misunderstanding what empty is for. It's meant to check that the size of a series/dataframe is greater than 0, meaning there are rows. For example,

df.iloc[1:0]

Empty DataFrame
Columns: [1, 2, 3]
Index: []

df.iloc[1:0].empty
True

If you want to check that a row has all NaNs, use isnull + all:

df.isnull().all(1)

0    False
1    False
2     True
3    False
4    False
5     True
6     True
dtype: bool

For your example, this should do:

df[2:3].isnull().all(1).item()
True

Note that you can't use item if your slice is more than one row in size.

answered Nov 16, 2017 at 16:19

cs95

406k106 gold badges744 silver badges797 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

MaxU - stand with Ukraine · Accepted Answer · 2017-11-16 16:21:38Z

2

I guess you are looking for something like this:

In [296]: df[5:]
Out[296]:
    1   2   3
5 NaN NaN NaN
6 NaN NaN NaN

In [297]: df[5:].isnull().all(1).all()
Out[297]: True

or even better (as proposed by @IanS):

In [300]: df[5:].isnull().all().all()
Out[300]: True

edited Nov 16, 2017 at 16:21

answered Nov 16, 2017 at 16:19

MaxU - stand with Ukraine

212k37 gold badges402 silver badges436 bronze badges

1 Comment

BENY Over a year ago

@MaxU I post in a wrong place ....haha , should post under the question ,.,, sorry

Alexander · Accepted Answer · 2017-11-16 16:27:04Z

2

You can drop all null values from your selection and check if the result is empty:

>>> df[5:].dropna(how='all').empty
True

answered Nov 16, 2017 at 16:27

Alexander

111k32 gold badges212 silver badges208 bronze badges

Comments

BENY · Accepted Answer · 2017-11-16 16:31:54Z

1

If you are do not want to count NaN value as real number , this will equal to

df.dropna().iloc[5:]

You select the line did not exist in your dataframe

df.dropna().iloc[5:].empty
Out[921]: True

answered Nov 16, 2017 at 16:31

BENY

324k22 gold badges176 silver badges250 bronze badges

Comments

Harikrishna · Accepted Answer · 2017-11-17 06:11:35Z

1

If you have a dataframe and want to drop all rows containing NaN in each of the columns, you can do this

df.dropna(how='all')

Noticed that your dataframe also has NaN in one the columns in some cases. If you need to drop the entire row in such case:

df.dropna(how='any')

After you do this (which ever is your preference) you could check length of dataframe (number of rows it contains) using:

len(df)

answered Nov 17, 2017 at 6:11

Harikrishna

1,1405 gold badges13 silver badges32 bronze badges

Comments

Dharman · Accepted Answer · 2020-01-26 22:10:44Z

1

I guess you have to use isnull() instead of empty().

import pandas 
df = pandas.DataFrame(columns=[1,2,3], data=[[1,2,3],[1,None,3],[None, None, None],[3,2,1],[4,5,6],[None,None,None],[None,None,None]])
df[2:3].isnull()
1   2   3
True    True    True

edited Jan 26, 2020 at 22:10

Dharman♦

33.9k27 gold badges106 silver badges157 bronze badges

answered Nov 16, 2017 at 16:22

Saghe Achraf

3401 silver badge7 bronze badges

1 Comment

cs95 Over a year ago

See stackoverflow.com/a/47334457/4909087 and stackoverflow.com/a/47334464/4909087

Collectives™ on Stack Overflow

Empty DataFrame doesn't admit its empty

6 Answers 6

Comments

1 Comment

Comments

Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

Comments

1 Comment

Comments

Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related