I have a Dataframe of a million rows. I am trying to break it up into small sized Dataframes of 1000 rows each. I am able to break this huge Dataframe into smaller chunks (of 1000 rows each) using the below code:
size = 1000
list_of_dfs = [df[i:i+size-1,:] for i in range(0, len(df),size)]
I am however unsure how to call each of these smaller Dataframes in a loop so that I can read the entire 1000 small Dataframes that have been created
Output is list of DataFrames, so you can loop them:
out = []
for df_small in list_of_dfs:
print (df_small)
#procesing...
out.append(df_small)
Similar:
for i in range(0, len(df),size):
df_small = df[i:i+size-1,:]
print (df_small)
If data are in csv is possible use chunksize parameter:
for df_small in pd.read_csv(filename, chunksize=size):
print (df_small)
list_of_dfs[0] and so on but I am not sure how many such dataframes exists. Hope I made my request clearer here.. Thanks.
list_of_dfs[0], list_of_dfs[1], list_of_dfs[N] then N is N = len(list_of_dfs-1)