looping the data using date variable in python

Question

I have the following dataframe with datetime, lon and lat variables. This data is collected for each second which means each date is repeated 60 times

I am doing some calculations using lat, lon values and at the end I need to write this data to Postgres table.

2016-07-27 06:43:45    50.62    3.15
2016-07-27 06:43:46    50.67    3.22
2016-07-28 07:23:45    52.32    3.34
2016-07-28 07:24:46    52.67    3.45

Currently I have 10 million records . It is taking longer time if I use whole dataframe for computing.

How can I loop this for each date, write it to DB and clear the dataframe??

I have converted the datetime variable to date format

df['date'] = df['datetime'].dt.date
df = df.sort(['datetime'])

my computation is
df.loc[(df['lat'] > 50.10) & (df['lat'] <= 50.62), 'var1'] = 1
df.loc[(df['lan'] > 3.00) & (df['lan'] <= 3.20), 'var2'] = 1

Writing it to DB

df.to_sql('Table1', engine,if_exists = "replace",index = False)

Muon · Accepted Answer · 2016-09-13 06:22:36Z

1

Have you considered using the groupby() function? You can use it to treat each 'date' as a seperate DataFrame and then run your computations.

for sub_df in df.groupby('date'):
    # your computations

answered Sep 13, 2016 at 6:22

Muon

1,3461 gold badge13 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

looping the data using date variable in python

1 Answer 1

Comments

Your Answer

Collectives™ on Stack Overflow

looping the data using date variable in python

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related