Average for large csv file

Question

suppose I have some data that looks like this:

time value
 1    1
 2    2
 3    3
 4    4

and it goes to 20,000, how would I get the average and stdev of this data?

1    import sys
2    import os
3    import string
4    import csv
5    import math
6    average = []
7    filename = 'pei1.rg'
8    fin1 = open(filename,"rb")
9    line1 = fin1.readline()
10   line1 = fin1.readline()
11   while line1:
12       line1 = map(string.strip,line1.strip().split())
13       average.append(float(line1[1]))
14       line1 = fin1.readline()
15   #print float(average)/float(count)
16   print "Number of steps: ", len(average)
17   print "Average over entire trajectory: ", float(sum(average))/float(len(average))
18   second = int(len(average)/2)
19   print "Average over second half: ", float(sum(average[second:]))/float(second)
20   sys.exit()

I think this works for calculating averages, but I do not know how to get the stdv from here.

What have you tried? Please post your code you have so far. If you haven't tried it yet, I would recommend reading the guidelines on questions on this site. StackOverflow is not a code-writing service. — Jake Griffin
– Jake Griffin, Commented Aug 20, 2015 at 22:22
i tried to post the code but it keeps telling me error code, I will try to post again. Sorry. — dennis
– dennis, Commented Aug 20, 2015 at 22:31

mhawke · Accepted Answer · 2015-08-20 23:05:59Z

1

In Python 3 you can use the statistics module, but first you need to arrange the values from your data into a sequence. Your data comes from a CSV file, so open the file, read in the lines, and convert the value to a float:

import statistics

with open('input.csv') as f:
    _ = next(f)    # skip the header line
    values = [float(line.split()[1]) for line in f]
    print('Average: {:.3f}'.format(statistics.mean(values)))
    print('Stdev: {:.3f}'.format(statistics.stdev(values)))

Output

Using the 4 values from your sample data:

Average: 2.500
Stdev: 1.291

It looks like you are using Python 2. There is a back port of the Python 3 statistics module for Python 2, and the above code will run without change. See https://pypi.python.org/pypi/statistics for info and install with pip.

edited Aug 20, 2015 at 23:05

answered Aug 20, 2015 at 22:46

mhawke

87.5k10 gold badges122 silver badges142 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

dennis Over a year ago

thanks, I am trying this out now, pretty new to programming so I am working out some kinks

dennis Over a year ago

it worked! Thanks and it is much simpler than my code.

mhawke Over a year ago

Glad to hear it. You might want to consider up-voting or even accepting this answer.

dennis Over a year ago

Done and done, however it says my upvote won't count until I've received 15 reputation. Hopefully it helped. Again, thanks!

Collectives™ on Stack Overflow

Average for large csv file

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related