Parsing csv file using datetime in python to identify unique dates

Question

I have a long list of weather variables across time that I have already filtered to remove those that don't conform to certain criteria. For example, all of the data points only lie between 11am (11) and 5pm (17pm). The data that lies between 11 and 17 o clock represents a single event, and not every day contains an event. I am trying to determine which days exhibited an event.

The data looks like this:

hd,Station Number,Year Month Day Hours Minutes in YYYY,MM,DD,HH24,MI format in Local time,Year Month Day Hours Minutes in YYYY,MM,DD,HH24,MI format in Local standard time,Year Month Day Hours Minutes in YYYY,MM,DD,HH24,MI format in Universal coordinated time,Precipitation since last (AWS) observation in mm,Quality of precipitation since last (AWS) observation value,Air Temperature in degrees Celsius,Quality of air temperature,Air temperature (1-minute maximum) in degrees Celsius,Quality of air temperature (1-minute maximum),Air temperature (1-minute minimum) in degrees Celsius,Quality of air temperature (1-minute minimum),Wet bulb temperature in degrees Celsius,Quality of Wet bulb temperature,Wet bulb temperature (1 minute maximum) in degrees Celsius,Quality of wet bulb temperature (1 minute maximum),Wet bulb temperature (1 minute minimum) in degrees Celsius,Quality of wet bulb temperature (1 minute minimum),Dew point temperature in degrees Celsius,Quality of dew point temperature,Dew point temperature (1-minute maximum) in degrees Celsius,Quality of Dew point Temperature (1-minute maximum),Dew point temperature (1 minute minimum) in degrees Celsius,Quality of Dew point Temperature (1 minute minimum),Relative humidity in percentage %,Quality of relative humidity,Relative humidity (1 minute maximum) in percentage %,Quality of relative humidity (1 minute maximum),Relative humidity (1 minute minimum) in percentage %,Quality of Relative humidity (1 minute minimum),Wind (1 minute) speed in km/h,Wind (1 minute) speed quality,Minimum wind speed (over 1 minute) in km/h,Minimum wind speed (over 1 minute) quality,Wind (1 minute) direction in degrees true,Wind (1 minute) direction quality,Standard deviation of wind (1 minute),Standard deviation of wind (1 minute) direction quality,Maximum wind gust (over 1 minute) in km/h,Maximum wind gust (over 1 minute) quality,Visibility (automatic - one minute data) in km,Quality of visibility (automatic - one minute data),Mean sea level pressure in hPa,Quality of mean sea level pressure,Station level pressure in hPa,Quality of station level pressure,QNH pressure in hPa,Quality of QNH pressure,#
    hd,40842,2000,3,22,13,40,2000,3,22,13,40,2000,3,22,13,40,0,N,20.4,N,20.5,N,20.4,N,20.2,N,20.2,N,20.1,N,20.1,N,20.1,N,20,N,98,N,,N,,N,9,N,8,N,18,N,7,N,11,N,,N,1013.3,N,1012.2,N,1013.3,N,#
    hd,40842,2000,3,22,13,47,2000,3,22,13,47,2000,3,22,13,47,0,N,20.5,N,20.5,N,20.5,N,20.2,N,20.2,N,20.2,N,20.1,N,20.1,N,20,N,97,N,,N,,N,4,N,0,N,56,N,75,N,5,N,,N,1013.2,N,1012.1,N,1013.2,N,#
    hd,40842,2000,3,23,11,0,2000,3,23,11,0,2000,3,23,11,0,0,N,23.4,N,23.4,N,23.3,N,21.3,N,21.4,N,21.3,N,20.2,N,20.3,N,20.2,N,82,N,,N,,N,8,N,5,N,66,N,2,N,9,N,,N,1013.6,N,1012.5,N,1013.6,N,#
    hd,40842,2000,3,23,11,1,2000,3,23,11,1,2000,3,23,11,1,0,N,23.4,N,23.4,N,23.4,N,21.4,N,21.4,N,21.3,N,20.3,N,20.3,N,20.2,N,82,N,,N,,N,8,N,5,N,68,N,3,N,9,N,,N,1013.6,N,1012.5,N,1013.6,N,#

The output file will ideally have the same format as the data shown above, but with only the lines that signify the start and end of unique events. This is my attempt at producing a code that will perform this task.

import csv
import datetime

with open("X:/weatherresults/final output/weather_out_2000_2006_time_filtered_and_speed_filtered.csv", "rb") as input, open("X:\weatherresults\sea_breeze_dates.csv", "wb") as wanted:
    reader = csv.DictReader(input, delimiter=",", skipinitialspace=True)
    fieldnames = reader.fieldnames
    writer_wanted = csv.DictWriter(wanted, fieldnames, delimiter=",")
    prev_row = None
    for line_number, row in enumerate(reader):
        try:
            dt = datetime.date(year=row["Year Month Day Hours Minutes in YYYY"], month=row["MM"], day=row["DD"])
            if prev_row is not None and dt > prev_row['dt']:
                writer_wanted.writerow(prev_row['row'])
                writer_wanted.writerow(row)
            prev_row = {'row':row, 'dt':dt}
        except:
            print "Failed to parse line", line_number
            print row

The code does not return any errors, but it always produces an exception. That is, it fails to parse every singe line, and the output file contains no data. Can anyone see what the error is in my code that is causing it to fail to parse every line?

runfile('X:/python/eventgrab.py', wdir='X:/python') File "X:/python/eventgrab.py", line 22 ^ IndentationError: unexpected unindent — Joss Kirk
– Joss Kirk, Commented Apr 4, 2016 at 2:36
but indenting that particular row (I just tried) does not fix the issue (after putting except back in — Joss Kirk
– Joss Kirk, Commented Apr 4, 2016 at 2:40

Don Kirkby · Accepted Answer · 2016-04-04 03:10:08Z

1

I think you're passing strings to the date() function. You need to convert the fields to integers with int().

Also, it might be simpler to group the lines by date using the groupby() function.

answered Apr 4, 2016 at 3:10

Don Kirkby

56.7k28 gold badges225 silver badges307 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Joss Kirk Over a year ago

You're a legend! That seems to have solved the issue. I'll have to peruse the output data to check everything worked. Thank you!

T. Arboreus · Accepted Answer · 2016-04-04 03:51:49Z

Superficially, your problem is with this line:

dt = datetime.date(year=row["Year Month Day Hours Minutes in YYYY"], month=row["MM"], day=row["DD"])

datetime.date takes integers, not strings. Something like this will fix your TypeError:

year = row["Year Month Day Hours Minutes in YYYY"]
month = row["MM"]
day = row['DD']
year = int(year)
month = int(month)
day = int(day)
dt = datetime.date(year=year,month=month,day=day)

The real problem is with your try/except statement. Because it's a blanket statement (i.e., doesn't refer to a particular class of errors), it will be impossible to get an error message that will let you debug your code. If you're getting parsing errors that you want to skip, use:

try <errorname>:
    ...

Collectives™ on Stack Overflow

Parsing csv file using datetime in python to identify unique dates

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related