Add date columns between 2 dates in Pandas dataframe

Question

I have an existing dataframe which looks like:

    id  start_date  end_date
0   1   20170601    20210531
1   2   20181001    20220930
2   3   20150101    20190228
3   4   20171101    20211031

I am trying to add 85 columns to this dataframe which are:

if the month/year (looping on start_date to end_date) lie between 20120101 and 20190101: 1
else: 0

I tried the following method:

start, end = [datetime.strptime(_, "%Y%m%d") for _ in ['20120101', '20190201']]
global_list = list(OrderedDict(((start + timedelta(_)).strftime(r"%m/%y"), None) for _ in range((end - start).days)).keys())

def get_count(contract_start_date, contract_end_date):
    start, end = [datetime.strptime(_, "%Y%m%d") for _ in [contract_start_date, contract_end_date]]
    current_list = list(OrderedDict(((start + timedelta(_)).strftime(r"%m/%y"), None) for _ in range((end - start).days)).keys())
    temp_list = []
    for each in global_list:
        if each in current_list:
            temp_list.append(1)
        else:
            temp_list.append(0)
    return pd.Series(temp_list)

sample_df[global_list] = sample_df[['contract_start_date', 'contract_end_date']].apply(lambda x: get_count(*x), axis=1)

and the sample df looks like:

customer_id contract_start_date contract_end_date   01/12   02/12   03/12   04/12   05/12   06/12   07/12   ... 04/18   05/18   06/18   07/18   08/18   09/18   10/18   11/18   12/18   01/19
1   1   20181001    20220930    0   0   0   0   0   0   0   ... 0   0   0   0   0   0   1   1   1   1
9   2   20160701    20200731    0   0   0   0   0   0   0   ... 1   1   1   1   1   1   1   1   1   1
3   3   20171101    20211031    0   0   0   0   0   0   0   ... 1   1   1   1   1   1   1   1   1   1
3 rows × 88 columns

it works fine for small dataset but for 160k rows it didn't stopped even after 3 hours. Can someone tell me a better way to do this?

Facing problems when the dates overlap for same customer.

What are you actually trying to do? This isn't the end goal is it? — Andy Hayden
– Andy Hayden, Commented Feb 7, 2019 at 20:04
@AndyHayden this is the end goal. Do you think this is possible? — bazinga
– bazinga, Commented Feb 7, 2019 at 20:06
@AndyHayden the end goal is to basically use it as a time series data after merging with some other dfs — bazinga
– bazinga, Commented Feb 7, 2019 at 20:08

Andy Hayden · Accepted Answer · 2019-02-07 20:57:20Z

First I'd cut off the dud dates, to normalize the end_time (to ensure it's in the time range):

In [11]: df.end_date = df.end_date.where(df.end_date < '2019-02-01', pd.Timestamp('2019-01-31')) + pd.offsets.MonthBegin()

In [12]: df
Out[12]:
   id start_date   end_date
0   1 2017-06-01 2019-02-01
1   2 2018-10-01 2019-02-01
2   3 2015-01-01 2019-02-01
3   4 2017-11-01 2019-02-01

Note: you'll need to do the same trick for start_date if there are dates prior to 2012.

I'd create the resulting DataFrame from a date range of the columns and then fill it in (with ones at start time and something else:

In [13]: m = pd.date_range('2012-01-01', '2019-02-01', freq='MS')

In [14]: res = pd.DataFrame(0., columns=m, index=df.index)

In [15]: res.update(pd.DataFrame(np.diag(np.ones(len(df))), df.index, df.start_date).groupby(axis=1, level=0).sum())

In [16]: res.update(-pd.DataFrame(np.diag(np.ones(len(df))), df.index, df.end_date).groupby(axis=1, level=0).sum())

The groupby sum is required if multiple rows start or end in the same month.

# -1 and NaN were really placeholders for zero
In [17]: res = res.replace(0, np.nan).ffill(axis=1).replace([np.nan, -1], 0)

In [18]: res
Out[18]:
   2012-01-01  2012-02-01  2012-03-01  2012-04-01  2012-05-01     ...      2018-09-01  2018-10-01  2018-11-01  2018-12-01  2019-01-01
0         0.0         0.0         0.0         0.0         0.0     ...             1.0         1.0         1.0         1.0         1.0
1         0.0         0.0         0.0         0.0         0.0     ...             0.0         1.0         1.0         1.0         1.0
2         0.0         0.0         0.0         0.0         0.0     ...             1.0         1.0         1.0         1.0         1.0
3         0.0         0.0         0.0         0.0         0.0     ...             1.0         1.0         1.0         1.0         1.0

it is not working properly if the dates for the same customer are overlapping. I have added an example for the same in the question.

Collectives™ on Stack Overflow

Add date columns between 2 dates in Pandas dataframe

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related