insert rows for missing values of datetime series in pandas dataframes

Question

I have some time series, recorded in 1 minute time steps without records, if the value of x is 0. The data file looks like this:

Date;Time;x 20.02.2020;00:00:00;0.1 20.02.2020;00:03:00;0.4 20.02.2020;00:04:00;0.3 20.02.2020;00:05:00;0.3 20.02.2020;00:07:00;0.2

I want to fill in the missing records having x=0.0. The expected result is:

Date;Time;x 20.02.2020;00:00:00;0.1 20.02.2020;00:01:00;0.0 20.02.2020;00:02:00;0.0 20.02.2020;00:03:00;0.4 20.02.2020;00:04:00;0.3 20.02.2020;00:05:00;0.3 20.02.2020;00:06:00;0.0 20.02.2020;00:07:00;0.2

I tried:

import pandas as pd

with open('data.csv') as csv_file:
    df = pd.read_csv(csv_file, delimiter=';',parse_dates={'datetime': ['Date', 'Time']})
df.set_index(['datetime'])
df.asfreq(freq='1Min', fill_value=0.0)

And I get:

datetime x 1970-01-01 1970-01-01 0.0

No error message. What's wrong?

Erfan · Accepted Answer · 2020-02-27 23:30:03Z

2

Because your set_index operation is not happening in place, best is to chain your methods:

df.set_index('datetime').asfreq('1min', fill_value=0).reset_index()

Or with resample:

df.set_index('datetime').resample('1min').first().fillna(0).reset_index()

             datetime    x
0 2020-02-20 00:00:00  0.1
1 2020-02-20 00:01:00  0.0
2 2020-02-20 00:02:00  0.0
3 2020-02-20 00:03:00  0.4
4 2020-02-20 00:04:00  0.3
5 2020-02-20 00:05:00  0.3
6 2020-02-20 00:06:00  0.0
7 2020-02-20 00:07:00  0.2

Or fix your own code:

df = df.set_index('datetime')
df.asfreq(freq='1Min', fill_value=0)

answered Feb 27, 2020 at 23:30

Erfan

43.4k10 gold badges76 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

insert rows for missing values of datetime series in pandas dataframes

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related