Read and write to the same csv file - delete row if condition met (stop writerow to erase everything in file)

Question

I am trying to create an archive to store a list of available books in the system. I want my program to ask the user to input csv file, read a list of books from that file, check the year of publication and delete the row if the book is older than 7 years. I want to keep everything in a single file.

So far, instead of deleting certain rows, writerow deletes everything in the file. Could someone help me to understand how to fix it?

import csv
import os
import time

archive = os.listdir()


def get_user_files(self):
    while True:
        for position, file_name in enumerate(archive):
            print(position, "-", file_name)
        userInput = input("\n\n ")
        if (int(userInput) < 0) or (int(userInput) > len(archive)):
            print("Invalid Input. Try again. \n")
        else:
            print("Loading succesful!")
            break

    global cvs_list
    cvs_list = archive[int(userInput)]  # Store file
    archive.remove(cvs_list)  # Remove from the list


    with open(cvs_list, 'r') as in_file, open(cvs_list, 'w') as out_file:
        reader = csv.reader(in_file)
        writer = csv.writer(out_file)
        for row in reader:
            next(reader) #skip headers
            if int(row[2]) < 2011:
                writer.writerow(row)

Edit:

with open(cvs_list, 'r') as in_file:
        csv_in = csv.reader(in_file, quoting=csv.QUOTE_ALL)
        filtered_list = []
        row1 = next(csv_in)
        filtered_list.append(row1)
        for row in csv_in:
            if int(row[2]) >= 2011: 
                row.append(filtered_list)

        with open(cvs_list, 'w') as out_file:
            writer = csv.writer(out_file)
            writer.writerows(filtered_list)

Ben · Accepted Answer · 2018-07-11 17:20:22Z

2

It's generally not advised to read and write to the same open file handle for reasons like this. Instead, read the entire file to a data structure, and in a separate with block, write your new data. This also makes it easier to write to a different file (perhaps with a timestamp attached), which can be handy when you (like everyone) inevitably screw something up and need to try your new code on your old data- you have a backup.

answered Jul 11, 2018 at 17:20

Ben

6,4834 gold badges38 silver badges46 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Micks Ketches · Accepted Answer · 2018-07-11 21:46:22Z

0

import csv

def filter_dates(csv_filepath)
    with open(csv_filepath, 'r') as in_file:
        csv_in = list(csv.reader(in_file))
    # create accumulator for new list to add only valid values
    filtered_list = []
    filetered_list.append(csv_in[0]) # append header to new list
    # filter the list making sure no errors appear BEFORE writing to the file
    for row in csv_in[1:]: #skip header (first row)
        if int(row[2]) >= 2011: # if entry is NEWER OR EQUAL TO than 7 years, we add it to filtered_list
            filtered_list.append(row)
    # now filtered_list contains only entires where index position 2 contains valid years
    with open(csv_filepath, 'w') as out_file:
        writer = csv.writer(out_file)
        writer.writerows

Here is a fully independent example:

 csv_in = [
    ['name', 'sbin', 'year'],
    ['moby_dick', 'sbin', '1851'],
    ['new_book', 'sbin', '2011'],
    ['newest_book', 'sbin', '2018'],

]
filtered_list = []
filtered_list.append(csv_in[0]) # this is where the header is added
for row in csv_in[1:]: #skip header (first row)
    if int(row[2]) >= 2011:
        filtered_list.append(row)
print(filtered_list)

A couple of notes:

it's generally good to store this kind of stuff in memory before you open the file to write (or overwrite in this case) so that any error while reading and filtering the file happens before we try to modify the output
easiest way to overwrite a file is to first read it, commit the contents to memory (csv_in the array I've defined in the first with block this case), and then *finally8 once the data is ready (filtered_list) for 'shipping' commit it to a file
never ever use the global declaration in python, it's never worth it and causes a lot of headaches down the line

test

edited Jul 11, 2018 at 21:46

answered Jul 11, 2018 at 17:43

Micks Ketches

4861 gold badge4 silver badges14 bronze badges

3 Comments

MagicCat Over a year ago

Hello, Thank you for your reply. Unfortunately, I can't make your code work for me. I tried to adjust the code, but I keep getting errors. The first error I get was 'local variable 'row' referenced before assignment'. Next error that occurs is for row in csv_in[1:]: TypeError: '_csv.reader' object is not subscriptable I have changed the code a bit and currently, it saves the headers but no other values are saved into the new file. Could you please guide me how to fix it? Thanks

Micks Ketches Over a year ago

I had a typo, it should be filtered_list.append(row)

Micks Ketches Over a year ago

@MagicCat I have also added an independent example, this should be a good reference.

dawg · Accepted Answer · 2018-07-11 22:49:20Z

You have:

with open(cvs_list, 'r') as in_file:
        csv_in = csv.reader(in_file, quoting=csv.QUOTE_ALL)
        filtered_list = []
        row1 = next(csv_in)
        filtered_list.append(row1)
        for row in csv_in:
            if int(row[2]) >= 2011: 
                row.append(filtered_list)
        # WRONG! you are opening the same file for output 
        # in the upper block
        with open(cvs_list, 'w') as out_file:
            writer = csv.writer(out_file)
            writer.writerows(filtered_list)

It is far better to read and write at the same time then copy the tmp file onto the source.

Like this:

#  NOT TESTED!
with open(cvs_list, 'r') as in_file, open(tmp_file, 'w') as out_file:
    csv_in = csv.reader(in_file, quoting=csv.QUOTE_ALL)
    writer = csv.writer(out_file)
    writer.writerow(next(csv_in))
    writer.writerows(row for row in csv_in if int(row[2])>=2011)

Then at the end of that with block you can copy the temp file on top of the source file:

from shutil import move
move(tmp_file, cvs_list)

Collectives™ on Stack Overflow

Read and write to the same csv file - delete row if condition met (stop writerow to erase everything in file)

3 Answers 3

Comments

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related