Reading CSV files in a loop using pandas, then concatenating them

Question

I have 10 csv files, named data_run1_all.csv, data_run2_all.csv, ..., data_run10_all.csv. CSV files have same columns, but different rows.

Now I am importing them one by one to df_run1, df_run2, ..., df_run10.

Can I use a loop to import them? Something like: i=1 to 10, df_runi=pandas.read_csv('data_runi_all.csv').

I am asking because the data analysis, plotting, etc. for each data frame are same, too. All the code for each data frame is repeated 10 times. If I can use a loop to do 10 times, the code will be much shorter and readable.

cs95 · Accepted Answer · 2017-09-30 13:19:43Z

10

Read your CSVs in a loop and call pd.concat:

file_name = 'data_run{}_all.csv'
df_list = []
for i in range(1, 11):
    df_list.append(pd.read_csv(file_name.format(i))

df = pd.concat(df_list)

Alternatively, you could build the list inside a comprehension:

file_name = 'data_run{}_all.csv'
df = pd.concat([pd.read_csv(file_name.format(i)) for i in range(1, 11)])

edited Sep 30, 2017 at 13:19

answered Sep 30, 2017 at 13:02

cs95

406k106 gold badges745 silver badges798 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Horia Coman · Accepted Answer · 2017-09-30 13:03:09Z

1

You need to make df_run a list. You could do something like this:

df_run = []
for i in range(1,10):
  df_run.append(pandas.read_csv('data_run{0}_all.csv'.format(i))
for df in df_run:
  // Do your processing

Or do everything in a single loop, and avoid having the list.

answered Sep 30, 2017 at 13:03

Horia Coman

8,8112 gold badges25 silver badges25 bronze badges

Collectives™ on Stack Overflow

Reading CSV files in a loop using pandas, then concatenating them

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related