Python Pandas dataframe shift does not work in apply functions

Question

I am getting this error below

AttributeError: ("'float' object has no attribute 'shift'", 'occurred at index 718170')

on running my pandas scripts below.

def volumediff(x):
    if x['positive_mvt'] == True:
        volume_d = x['volume'].shift(1)
    else:
        volume_d = ""
    return volume_d

df['new_volume'] = df.apply(volumediff,axis=1)

So because of this I believe based on a almost similar error at AttributeError: 'float' object has no attribute 'split', I thought the issue is caused by a null value since the shift function takes the value that might be out of my dataset. However, I was successful in doing the below without having any issue.

df['new_volume'] = df['volume'].shift(1)

Unfortunately it just doesn't work with an apply function, which I need because I need to use "if else".

I have tried to get around by using the script below - by using an try except to skip any cells which create a value issue. But I am receiving "NA" and "" for all the values in my column, which shouldn't be the case.

def volumediff(x):
    if x['positive_mvt'] == True:
        try:
            volume_d = x['volume'].shift(1)
        except:
            volume_d = "NA"
    else:
        volume_d = ""
    return volume_d

df['new_volume'] = df.apply(volumediff,axis=1)

Original sample df:

x = [
    [False, 240.20353],
    [False, 621.28854],
    [True, 64.85972],
    [True, 151.86484],
    [False, 190.91042],
    [True, 128.78566],
    [False, 415.53138],
    [True, 43.14669],
    [True, 512.03531],
    [True, 502.41939],
]

df = pd.DataFrame(x, columns=['positive_mvt', 'volume'])

df
Out[1]: 
   positive_mvt     volume
0         False  240.20353
1         False  621.28854
2          True   64.85972
3          True  151.86484
4         False  190.91042
5          True  128.78566
6         False  415.53138
7          True   43.14669
8          True  512.03531
9          True  502.41939

Error example:

I checked my dataframe and I am suspecting that the issue might be caused by the conflict between my if function which only selects rows that are true, however some of the rows which are false are required by x[volume].shift(1) which is the row above it. But that wasn't the case because when I tried the script below, it wasn't working either and triggers the same attribute error. Looks like using the apply function just doesn't work with .shift.

def volumediff(x):
    volume_d = x['volume'].shift(1)
    return volume_d

df['new_volume'] = df.apply(volumediff,axis=1)

Anyone has any insights into how to solve this issue without doing two separately columns and sequentially work on on if else and the minus shift formula separately?

Just from what you've provided, I don't think you need apply here. Can you share a small sample of your frame and a desired output? — user3483203
– user3483203, Commented Jul 25, 2019 at 17:06
When you use df.apply(axis=1) you pass a Series to your function, (x in your code). It doesn't really make sense to shift this because now when you .shift you're instead shifting along the columns in the original DataFrame. On top of that, since x is a Series, when you do x['volume'] you now have a single item, in this case a float, which clearly has no property shift because what would it mean to "shift" the number 240.20353? — ALollz
– ALollz, Commented Jul 25, 2019 at 17:15

CJR · Accepted Answer · 2019-07-25 17:17:42Z

1

When you run apply it passes each column/row (in your case row) to the function that you're applying. If you call .shift() on a series it makes sense - you're shifting the series. Calling shift on a single value in your series, as you are doing, makes no sense (how do you shift 12? what does that even mean?).

What you want to be doing is this:

df['new_volume'] = df['volume'].shift(1)
df.loc[df['positive_mvt'] == False, 'new_volume'] = ""

Also I have no idea what your dtypes are and you should probably be careful with that.

answered Jul 25, 2019 at 17:17

CJR

3,9872 gold badges13 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

rrcal Over a year ago

And this is fast too. About 18 ns for a a df with 500,000 rows.

编程好Len难 · Accepted Answer · 2022-02-14 14:42:07Z

0

i find a way to avoid this problem. you can use groupby first before use apply functions(but you need to import something for groupby)

def shoupanjia1 (x,sp):

    return (x['收盘价'].shift(sp) - x['收盘价']) / x['收盘价']

w_df['下一周涨跌幅'] =w_df.groupby('名称').apply(shoupanjia1,sp=-1).values

answered Feb 14, 2022 at 14:42

编程好Len难

11 bronze badge

Collectives™ on Stack Overflow

Python Pandas dataframe shift does not work in apply functions

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related