1

in my code I've generated a range of dates using pd.date_range in an effort to compare it to a column of dates read in from excel using pandas. The generated range of dates is refered to as "all_dates".

all_dates=pd.date_range(start='1998-12-31', end='2020-06-23')
    
for i, date in enumerate(period):   # where 'Period' is the column of excel dates 
        if date==all_dates[i]:        # loop until date from excel doesn't match date from generated dates 
            continue
        else:
            missing_dates_stock.append(i)  # keep list of locations where dates are missing 
            stock_data.insert(i,"NaN")     # insert 'NaN' where missing date is found

This results in TypeError: argument of type 'Timestamp' is not iterable. How can I make the data types match such that I can iterate and compare them? Apologies as I am not very fluent in Python.

1
  • Are you looking for a way to add a NaN row if the date does not exist between 1998-12-31 and 2020-06-23? Commented Mar 30, 2021 at 0:58

1 Answer 1

2

I think you are trying to create a NaN row if the date does not exist in the excel file.

Here's a way to do it. You can use the df.merge option.

I am creating df1 to simulate the excel file. It has two columns sale_dt and sale_amt. If the sale_dt does not exist, then we want to create a separate row with NaN in the columns. To ensure we simulate it, I am creating a date range from 1998-12-31 through 2020-06-23 skipping 4 days in between. So we have a dataframe with 4 missing date between each two rows. The solution should create 4 dummy rows with the correct date in ascending order.

import pandas as pd
import random

#create the sales dataframe with missing dates

df1 = pd.DataFrame({'sale_dt':pd.date_range(start='1998-12-31', end='2020-06-23', freq='5D'),
                    'sale_amt':random.sample(range(1, 2000), 1570)
                })
                
print (df1)

#now create a dataframe with all the dates between '1998-12-31' and '2020-06-23'

df2 = pd.DataFrame({'date':pd.date_range(start='1998-12-31', end='2020-06-23', freq='D')})

print (df2)

#now merge both dataframes with outer join so you get all the rows.
#i am also sorting the data in ascending order so you can see the dates
#also dropping the original sale_dt column and renaming the date column as sale_dt
#then resetting index 

df1 = (df1.merge(df2,left_on='sale_dt',right_on='date',how='outer')
          .drop(columns=['sale_dt'])
          .rename(columns={'date':'sale_dt'})
          .sort_values(by='sale_dt')
          .reset_index(drop=True))


print (df1.head(20))

The original dataframe was:

        sale_dt  sale_amt
0    1998-12-31      1988
1    1999-01-05      1746
2    1999-01-10      1395
3    1999-01-15       538
4    1999-01-20      1186
...         ...       ...
1565 2020-06-03       560
1566 2020-06-08       615
1567 2020-06-13       858
1568 2020-06-18       298
1569 2020-06-23      1427

The output of this will be (first 20 rows):

    sale_amt    sale_dt
0     1988.0 1998-12-31
1        NaN 1999-01-01
2        NaN 1999-01-02
3        NaN 1999-01-03
4        NaN 1999-01-04
5     1746.0 1999-01-05
6        NaN 1999-01-06
7        NaN 1999-01-07
8        NaN 1999-01-08
9        NaN 1999-01-09
10    1395.0 1999-01-10
11       NaN 1999-01-11
12       NaN 1999-01-12
13       NaN 1999-01-13
14       NaN 1999-01-14
15     538.0 1999-01-15
16       NaN 1999-01-16
17       NaN 1999-01-17
18       NaN 1999-01-18
19       NaN 1999-01-19
Sign up to request clarification or add additional context in comments.

2 Comments

Worked like a charm, thank you! Explained it very well too.
glad i could be of help. Yes, the documentation is important so everyone can understand. Since this is going to be useful for others, it is important that we document code to help future SO members to understand what's been done in the code. Happy coding.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.