Error while merging multiple csv with a condition

Question

Requirement: I have 69 csvs. Merge them on the basis of below condition:

Condition: If value in cell at header 'col7' = 1, append the corresponding row in new csv.

Can you please help me complete this piece of code?

Below is my code:

with open('merged.csv', 'a') as mergedFile:
    for csv in glob('*.csv'):
        if csv == 'merged.csv':
            pass
        else:
            for line in os.listdir():
                for eachFile in open(csv, 'r'):
                    # write further code here if header 'col7' value = 1
                        # write further code here to add the corresponding rows meeting condition
                        mergedFile.write(line)

If there is any way of doing this by pandas, most welcome.

the name of the csv file is merged? what is the unique feature in the names of the csv? Also, do u know the names of ur headers? if u do, the dict form in the csv module might be helpful — sammywemmy
– sammywemmy, Commented Feb 29, 2020 at 4:40
The new file to which I am wriiting to is "merged.csv". Unique feature is of no concern here, this is more of like filtering data. And header name which I want to put filter over is 'col7' — Vers
– Vers, Commented Feb 29, 2020 at 14:26

tdelaney · Accepted Answer · 2020-02-29 05:24:06Z

The csv module is good for this task. You are mostly just filtering row by row. pandas would have to bring the whole csv into memory before writing.

import csv

# todo: do you want append, or use 'w' to start a new file?
with open('merged.csv', 'a', newline='') as mergedFile:
    writer = csv.writer(mergedFile)
    for csvFile in glob('*.csv'): # renamed variable to avoid module name collision
        if csvFile == 'merged.csv':
            continue
        with open(csvFile, newline='') as inFile:
            reader = csv.reader(inFile)
            # assuming there is a header with column names, we are looking for "col7"
            header = next(reader)
            try:
                filterCol = header.index("col7")
            except ValueError as e:
                print("no 'col7' in {}, skipping".format(csvFile))
                continue
            writer.writerows(row for row in reader if row[filterCol == "1")

Collectives™ on Stack Overflow

Error while merging multiple csv with a condition

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related