This is a doubt on an existing or closed thread, however I was disabled from adding a comment that's why this. I was checking out this code and wondering how this piece is carefully getting rid of headers from other files while merging ? I do not see any element in the code that is set to ignore the headers from the files. Code that I'm referring to:
Concatenating multiple csv files into a single csv with the same header - Python
import shutil
import glob
#import csv files from folder
path = r'data/US/market/merged_data'
allFiles = glob.glob(path + "/*.csv")
allFiles.sort() # glob lacks reliable ordering, so impose your own if output order matters
with open('someoutputfile.csv', 'wb') as outfile:
for i, fname in enumerate(allFiles):
with open(fname, 'rb') as infile:
if i != 0:
infile.readline() # Throw away header on all but first file
# Block copy rest of file from input to output without parsing
shutil.copyfileobj(infile, outfile)
print(fname + " has been imported.")
infile.readline()effectively discards the line so thatshutil.copyfileobjwill no longer read it when reading frominfile. This happens on all files except for the first (hence thei != 0, so that the output file will only have the header from the first file.