Reading CSV file results in list index out of range

Question

I have a csv file with numerous columns and some columns have more rows than others. I am trying to read certain columns and extract those columns to a different csv file. However, I keep getting list index out of range because the columns I'm reading are shorter than the other columns in the csv file.

import csv
import os
import os.path

'Open csv directory to extract data from'
path = 'S://Raw Data/VehicleData/'               

'Get file names and print number of files in the directory'
files = [f for f in os.listdir(path)]
print(files)
print(len(files))

'Start loop to scan through files in directory'
for j in range(len(files)):

'determine name of file and the output name of the file'
name = files[j]
print(name)

'path to file to open'
fpath = os.path.join(path, files[j])

    if files[j].endswith(".csv"):
        nameout = 'VehicleN3STPData' + '.csv'
        with open (fpath, 'r', newline='') as csvfile:
            testcsv = csv.reader(csvfile,delimiter=',')
            with open (nameout,'a+',newline='') as outfile:
                writer = csv.writer(outfile)
                for row in testcsv:
                    my_row = []
                    my_row.append(row[59])
                    my_row.append(row[111])
                    my_row.append(row[120])
                    print(my_row)
                    writer.writerow(my_row)
                outfile.close()    
        csvfile.close()

miketery · Accepted Answer · 2017-11-09 00:30:25Z

1

You need a conditional statement to check for length or you need a try & except clause depending on your intended behavior.

Here's an example of using a conditional statement that preserves the column values as blanks if they don't exist.

Instead of:

my_row.append(row[59]) 
my_row.append(row[111]) 
my_row.append(row[120])

Do:

for col_num in [59, 111, 120]:
    my_row.append(row[col_num] if len(row) > col_num else '')

edited Nov 9, 2017 at 0:30

answered Nov 9, 2017 at 0:22

miketery

1212 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Cody Howard · Accepted Answer · 2017-11-09 18:13:50Z

Thanks for your help, here is the final code that worked, removing the header with islice, using your conditional statement, and then removing the blank lines with if any(my_row) thanks!!

    if files[j].endswith(".csv"):
    nameout = 'VehicleN3STPData' + '.csv'
    with open (fpath, 'r', newline='') as csvfile:
        testcsv = csv.reader(csvfile,delimiter=',')
        with open (nameout,'a+',newline='') as outfile:
            writer = csv.writer(outfile)
            for row in islice(testcsv,3,None):
                    my_row = []
                    for col_num in [59,111,120]:
                        my_row.append(row[col_num] if len(row) > col_num else '')
                    if any(my_row):
                        writer.writerow(my_row)
            outfile.close()    
    csvfile.close()enter code here

Collectives™ on Stack Overflow

Reading CSV file results in list index out of range

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related