combining multiple files into a single file with DataFrame

Question

I have been able to generate several CSV files through an API. Now I am trying to combine all CSV's into a unique Master file so that I can then work on it. But it does not work. Below code is what I have attempted What am I doing wrong?

import glob
import pandas as pd
from pandas import read_csv

master_df = pd.DataFrame()

for file in files:
    df = read_csv(file)
    master_df = pd.concat([master_df, df])
    del df

master_df.to_csv("./master_df.csv", index=False)

It goes through but it does not combine the files into a Master one for some reason@ttreis — oss
– oss, Commented Jul 6, 2020 at 21:59
Weird, it creates the file for me. Do you have write permissions for the folder you want to write the file to? Did you try to print the variable first to see if only the writing fails? — ttreis
– ttreis, Commented Jul 6, 2020 at 22:01
Works for me, can not reproduce. Do all the csv files have the same format? Do they have headers? — wwii
– wwii, Commented Jul 6, 2020 at 22:20

sebbit · Accepted Answer · 2020-07-06 22:06:35Z

1

Although it is hard to tell what the precise problem is without more information (i.e., error message, pandas version), I believe it is that in the first iteration, master_df and df do not have the same columns. master_df is an empty DataFrame, whereas df has whatever columns are in your CSV. If this is indeed the problem, then I'd suggest storing all your data-frames (each of which represents one CSV file) in a single list, and then concatenating all of them. Like so:

import pandas as pd

df_list = [pd.read_csv(file) for file in files]

pd.concat(df_list, sort=False).to_csv("./master_df.csv", index=False)

Don't have time to find/generate a set of CSV files and test this right now, but am fairly sure this should do the job (assuming pandas version 0.23 or compatible).

answered Jul 6, 2020 at 22:06

sebbit

1213 silver badges4 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

combining multiple files into a single file with DataFrame

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related