Python: faster normalization

Question

I want to normalize my ambient temperature column (Ta).
Here is my code:

df['Ta'] = df['Ta'].apply(lambda v: (v - df['Ta'].min())) / (df['Ta'].max() - df['Ta'].min())

It works well. But, it is very slow. The file size is 20 MB with the shape of (300000, 8).

Is there any other faster solution to this?

Maybe you can cache df['Ta'].min() and df['Ta'].max() in variables, instead of recomputation every time the lambda is called? — Anthony Kong
– Anthony Kong, Commented Jul 15, 2019 at 22:12
actualy you have constants : df['Ta'], (df['Ta'].max() - df['Ta'].min()), df['Ta'].min() and do not modify the content of df['Ta'] , make another_array = df_ta.apply... — user8426627
– user8426627, Commented Jul 15, 2019 at 22:16
Thanks. I am not sure how to implement it. The following is correct for small data and works fast with the large data. But not sure whether it is a true solution. df['Ta'] = (df['Ta'] - df['Ta'].min()) / (df['Ta'].max() - df['Ta'].min()) — k.ko3n
– k.ko3n, Commented Jul 15, 2019 at 22:40

BENY · Accepted Answer · 2019-07-16 00:24:30Z

1

Since you are not take advantage of pandas , apply here is another layout of for loop , which will slow down the whole process

 import pandas as pd ; import numpy as np

 df['Ta']- df['Ta'].min() / np.ptp(df['Ta']))

answered Jul 16, 2019 at 0:24

BENY

324k22 gold badges176 silver badges250 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Parijat Bhatt · Accepted Answer · 2019-07-15 23:50:39Z

0

I am not sure if there would be a faster way:

mx = df['Ta'].max()
mn = df['Ta'].min()

df['Ta'] -=mn
df['Ta']/=(mx-mn)

answered Jul 15, 2019 at 23:50

Parijat Bhatt

6744 silver badges6 bronze badges