How to apply math functions while filtering (using loc) in pandas

Question

I have retrived a value of x as x = 10.64589195722904 which I need to match in my existing dataframe using loc. As subtraction result can be a negative value which I must ignore and so I am using math.fabs to achieve it.

fdf = df.loc[(math.fabs(df['x'] - x) <= 0.01 & math.fabs(df['x'] - x) >= 0.001)]

But this is throwing error:

TypeError                                 Traceback (most recent call last)
<ipython-input-256-b8a71a5bd17c> in <module>
     10 #     fdf = df.loc[math.fabs((df['x'] - k) <= 0.001) & (math.fabs(df['x'] - k) >= 0.0001) ]
     11 
---> 12 df.loc[(math.fabs(df['x'] - x) <= 0.01 & math.fabs(df['x'] - x) >= 0.001)]
     13 fdf.head()

~\.conda\envs\pyenv\lib\site-packages\pandas\core\series.py in wrapper(self)
    110         if len(self) == 1:
    111             return converter(self.iloc[0])
--> 112         raise TypeError(f"cannot convert the series to {converter}")
    113 
    114     wrapper.__name__ = f"__{converter.__name__}__"

TypeError: cannot convert the series to <class 'float'>

jezrael · Accepted Answer · 2020-09-30 05:25:35Z

1

Use numpy.fabs for processing values vectorized way and also add () around masks because priority operators:

s = np.fabs(df['x'] - x)
fdf = df[(s <= 0.01) & (s >= 0.001)]

Alternative is use Series.between:

fdf = df[np.fabs(df['x'] - x).between(0.01, 0.001)]

edited Sep 30, 2020 at 5:25

answered Sep 30, 2020 at 5:19

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

TrickOrTreat Over a year ago

This is awesome. Thanks mate :)

Trenton McKinney · Accepted Answer · 2020-09-30 05:26:11Z

1

math.fabs only takes a single value, so .apply can be used to create a new column, and then perform the Boolean selection.
As shown by jezrael, np.fabs can be used for a vectorized approach
- The benefit is, numpy is faster

# apply math.fabs and create a column
df['fabs'] = df['x'].apply(lambda row: math.fabs(row) - x)

# filter on the new column
fdf = df[(df['fabs'] <= 0.01) & (df['fabs'] >= 0.001)]

edited Sep 30, 2020 at 5:26

answered Sep 30, 2020 at 5:20

Trenton McKinney

63.2k41 gold badges169 silver badges212 bronze badges

Collectives™ on Stack Overflow

How to apply math functions while filtering (using loc) in pandas

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related