adding prefix to pandas column

Question

I'm trying to add a prefix to a DataFrame in pandas. It supposes to be very easy:

import pandas as pd
a=pd.DataFrame({
    'x':[1,2,3],
})
#this one works;
"mm"+a['x'].astype(str)
0    mm1
1    mm2
2    mm3
Name: x, dtype: object

But surprisingly, if I want to use a prefix of single letter 'm', it stops working:

#this one doesn't work
"m"+a['x'].astype(str)
TypeError                                 Traceback (most recent call last)
<ipython-input-21-808db8051ebc> in <module>
      1 #this one doesn't work
----> 2 "m"+a['x'].astype(str)

C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\ops\__init__.py in wrapper(left, right)
   1014             if is_scalar(right):
   1015                 # broadcast and wrap in a TimedeltaIndex
-> 1016                 assert np.isnat(right)
   1017                 right = np.broadcast_to(right, left.shape)
   1018                 right = pd.TimedeltaIndex(right)

TypeError: ufunc 'isnat' is only defined for datetime and timedelta.

So my questions are:

How to solve the problem?
What happened, it seems pandas is trying to do something fancy?
Why is 'm' is so special? (it seems other single letters are ok, e.g. 'b').

Dani Mesejo · Accepted Answer · 2019-11-15 16:59:46Z

3

The problem is that "m" is interpreted as TimeDelta:

from pandas.core.dtypes.common import is_timedelta64_dtype

print(is_timedelta64_dtype("m"))

Output

True

The function is_timedelta64_dtype is called when you do:

res = "m" + a['x'].astype(str)

Code (pandas)

elif is_timedelta64_dtype(right):
    # We should only get here with non-scalar or timedelta64('NaT')
    #  values for right
    # Note: we cannot use dispatch_to_index_op because
    #  that may incorrectly raise TypeError when we
    #  should get NullFrequencyError
    orig_right = right
    if is_scalar(right):
        # broadcast and wrap in a TimedeltaIndex
        assert np.isnat(right)
        right = np.broadcast_to(right, left.shape)
        right = pd.TimedeltaIndex(right)

Given that the value is an scalar also, it checks if it is NaT,

assert np.isnat(right)

Which is what triggers the Exception. A simple workaround is to put "m" inside a list:

res = ["m"] + a['x'].astype(str)
print(res)

Output

0    m1
1    m2
2    m3
Name: x, dtype: object

answered Nov 15, 2019 at 16:59

Dani Mesejo

62.2k6 gold badges57 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

ALollz Over a year ago

I'm guessing that probably shouldn't be the desired behavior. Do you want to open a bug report?

milos.ai · Accepted Answer · 2019-11-15 17:05:24Z

1

Solve the problem by changing to:

import numpy as np
np.array('m')+a['x'].astype(str)

For some reason pandas thinks this "m" marks time. Please check explanation from @Daniel Mesejo

edited Nov 15, 2019 at 17:05

answered Nov 15, 2019 at 16:50

milos.ai

3,9287 gold badges34 silver badges33 bronze badges

Comments

Karthick Mohanraj · Accepted Answer · 2019-11-15 16:52:45Z

0

Well this seems to be an issue with the python frontend interface. This might occur due to some conflicts while using a Spyder interface or a Jupyter notebook. I got the same error while running the code on Spyder. The issue got resolved when I used the same code by invoking python in the command line terminal instead of SPYDER OR Jupyter.

Try running this same code in the command line terminal by invoking the python command and it should work perfectly.

answered Nov 15, 2019 at 16:52

Karthick Mohanraj

1,6883 gold badges18 silver badges29 bronze badges

Collectives™ on Stack Overflow

adding prefix to pandas column

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related