Join multiple csv files from a folder into a single csv python

Question

i have around 100 csv files in a folder.

/path/to/directory/*.csv
 it has files abc.csv,dsf.csv,rgfb.csv.....etc

a view of csv file.

182 a   1   4   242 52450
182 a   1   2   242 7176
182 c   1   1   242 7176
182 c   1   1   242 7410

i want to take all this csv from the directory and make it in one csv. There are no column names, but all the csv has same no of columns(i.e 5), and i want to join all csv, and put it in pandas dataframe and give column names as

col1  col2  col3  col4  col5
data  data  data  data  data
...   ...    ...   ...   ...

what i tried was.

import os
csv_list = []
for root, dirs,files in os.walk("path/to/directory", topdown=True):
for name in files:
    csv_list.append(os.path.join(root, name))

i got the list of csv

then i did

import pandas as pd
combined_csv = pd.append( [ pd.read_csv(f) for f in csv_list ] )

but it is appending horizontally and not vertically.

Also i have to give column names to 'combined_csv' is there any better way?

jezrael · Accepted Answer · 2016-12-23 09:21:07Z

3

I think you need concat with parameter axis=1 if need append vertically:

combined_csv = pd.concat([ pd.read_csv(f, header=None) for f in csv_list ], axis=1)

And if need append horizontally is default parameter axis=0 which can be omit:

import pandas as pd
combined_csv = pd.concat([ pd.read_csv(f, header=None) for f in csv_list ], ignore_index=True)

If need set column names use parameter names:

names = ['col1','col2','col3','col4','col5']
combined_csv = pd.concat([ pd.read_csv(f, header=None, names = names) for f in csv_list ],
                           ignore_index=True)

edited Dec 23, 2016 at 9:21

answered Dec 23, 2016 at 7:28

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

12 Comments

Shubham R Over a year ago

check edit , i have a view of my csv file, and your answer is not working. i am getting 5 rows × 66 columns as i tried it for 11 csv files

Shubham R Over a year ago

1csv has 5 coulmns and while putting it to dataframe it is adding index, so 6 columns and concatenating 11 csv files it is giving me 11x6 = 66 columns

jezrael Over a year ago

I think I see problem, need parameter header=None

Shubham R Over a year ago

still, after adding parameter and running combined_csv = pd.concat([ pd.read_csv(f, header=None) for f in csv_list ], axis=1) i am getting 5 rows × 66 columns

jezrael Over a year ago

sure, five me a sec.

|

Collectives™ on Stack Overflow

Join multiple csv files from a folder into a single csv python

1 Answer 1

12 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

12 Comments

Your Answer

Sign up or log in

Post as a guest

Related