0

I have 30 csv files. I want to give it as input in for loop, in pandas?

Each file has names such as fileaa, fileab,fileac,filead,....

And i would like to receive one output.

Usually i use read_csv but due to memory error, 'read_csv' doesn't work.

f = "./file.csv"
df = pd.read_csv(f, sep="/", header=0, dtype=str)

P.S only the first file has the column title but number of columns are same

Improve this question
12
  • stackoverflow.com/questions/42931068/…, is this what you are looking for? Commented Jul 6, 2017 at 15:25
  • Details please: how many columns does each file have? how do you want the output formatted (perhaps with a small example)? what is the backtrace for the error you get? is file.csv one of the thirty files? Commented Jul 6, 2017 at 15:25
  • @Vinay exactly ! thank you ! i am trying the same code to see if it works ! i will let you know if "low_memory=True" flag works Commented Jul 6, 2017 at 15:27
  • @Nick each file has around 50 rows and numerous lines (5gb data) Commented Jul 6, 2017 at 15:28
  • 1
    This will allow you to read the csv file in chunks. Try playing around with the parameter. Maybe chunksize=500000, and if that's too much, bump it down. Commented Jul 7, 2017 at 13:55
Related questions
11
Reading Multiple CSV Files into Python Pandas Dataframe
1
Combining multiple CSV file into a single one
1
Reading multiple csv files and writing it to another csv file
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.