How can I tell if a dataframe is of mixed type?

Question

I want to assign values to the diagonal of a dataframe. The fastest way I can think of is to use numpy's np.diag_indices and do a slice assignment on the values array. However, the values array is only a view and ready to accept assignment when a dataframe is of a single dtype

Consider the dataframes d1 and d2

d1 = pd.DataFrame(np.ones((3, 3), dtype=int), columns=['A', 'B', 'C'])
d2 = pd.DataFrame(dict(A=[1, 1, 1], B=[1., 1., 1.], C=[1, 1, 1]))

d2

   A    B  C
0  1  1.0  1
1  1  1.0  1
2  1  1.0  1

Then let's get our indices

i, j = np.diag_indices(3)

d1 is of a single dtype and therefore, this works

d1.values[i, j] = 0
d1

   A  B  C
0  0  1  1
1  1  0  1
2  1  1  0

But not on d2

d2.values[i, j] = 0
d2

   A    B  C
0  1  1.0  1
1  1  1.0  1
2  1  1.0  1

I need to write a function and make it fail when df is of mixed dtype. How do I test that it is? Should I trust that if it is, this assignment via the view will always work?

You inspect d1.dtypes which is a Series itself, and you check if all have the same value. — willeM_ Van Onsem
– willeM_ Van Onsem, Commented Oct 17, 2017 at 15:52

Zero · Accepted Answer · 2017-10-17 16:13:00Z

14

You could use internal _is_mixed_type method

In [3600]: d2._is_mixed_type
Out[3600]: True

In [3601]: d1._is_mixed_type
Out[3601]: False

Or, check unique dtypes

In [3602]: d1.dtypes.nunique()>1
Out[3602]: False

In [3603]: d2.dtypes.nunique()>1
Out[3603]: True

A bit of de-tour, is_mixed_type checks how blocks are consolidated.

In [3618]: len(d1.blocks)>1
Out[3618]: False

In [3619]: len(d2.blocks)>1
Out[3619]: True

In [3620]: d1.blocks    # same as d1.as_blocks()
Out[3620]:
{'int32':    A  B  C
 0  0  1  1
 1  1  0  1
 2  1  1  0}

In [3621]: d2.blocks
Out[3621]:
{'float64':      B
 0  1.0
 1  1.0
 2  1.0, 'int64':    A  C
 0  1  1
 1  1  1
 2  1  1}

edited Oct 17, 2017 at 16:13

answered Oct 17, 2017 at 16:05

Zero

77.4k22 gold badges153 silver badges153 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Bharath M Shetty Over a year ago

I have seen this so many times in the source code :) though couldnt come up with this answer +1

Ralph Over a year ago

I'm trying to use the method is_mixed_type, but it says it's not defined. Has this changed in more recent Python versions?

galaxyan · Accepted Answer · 2017-10-17 15:57:58Z

3

def check_type(df):
  return len(set(df.dtypes)) == 1

or

 def check_type(df):
   return df.dtypes.nunique() == 1

edited Oct 17, 2017 at 15:57

answered Oct 17, 2017 at 15:57

galaxyan

6,1593 gold badges23 silver badges44 bronze badges

2 Comments

willeM_ Van Onsem Over a year ago

A set(..) is never equal to 1. It can be equal to {1} (although not here). You probably want len(set(df.dtypes)).

galaxyan Over a year ago

@WillemVanOnsem should be len

willeM_ Van Onsem · Accepted Answer · 2017-10-17 15:54:40Z

1

You can inspect DataFrame.dtypes to check the types of the columns. For instance:

>>> d1.dtypes
A    int64
B    int64
C    int64
dtype: object
>>> d2.dtypes
A      int64
B    float64
C      int64
dtype: object

Given that there is at least one column, you can thus check this with:

np.all(d1.dtypes == d1.dtypes[0])

For your dataframes:

>>> np.all(d1.dtypes == d1.dtypes[0])
True
>>> np.all(d2.dtypes == d2.dtypes[0])
False

You can of course first check whether there is at least one column. So we can construct a function:

def all_columns_same_type(df):
    dtypes = df.dtypes
    return not dtypes.empty and np.all(dtypes == dtypes[0])

answered Oct 17, 2017 at 15:54

willeM_ Van Onsem

482k33 gold badges483 silver badges624 bronze badges

Collectives™ on Stack Overflow

How can I tell if a dataframe is of mixed type?

3 Answers 3

2 Comments

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related