python - how to count number of numbers from columns?

Question

I have a file with the following input data:

       IN   OUT
data1  2.3  1.3
data2  0.1  2.1
data3  1.5  2.8
dataX  ...  ...

There are thousands of such files and each has the same data1, data2, data3, ..., dataX I'd like to count number of each value for each data and column from all files. Example:

In file 'data1-IN' (filename)

2.3 - 50    (times)
0.1 - 233   (times)
... - ...   (times)

In file 'data1-OUT' (filename)

2.1 - 1024 (times)
2.8 - 120  (times)
... - ...  (times)

In file 'data2-IN' (filename)

0.4 - 312    (times)
0.3 - 202   (times)
... - ...   (times)

In file 'data2-OUT' (filename)

1.1 - 124 (times)
3.8 - 451  (times)
... - ...  (times)

In file 'data3-IN' ...

Which Python data structure would be the best to count such data ? I wanted to use multidimensional dictionary but I am struggling with KeyErrors etc.

Martijn Pieters · Accepted Answer · 2012-10-02 20:07:15Z

3

You really want to use collections.Counter, perhaps contained in a collections.defaultdict:

import collections
import csv

counts = collections.defaultdict(collections.Counter)

for filename in files:
    for line in csv.reader(open(filename, 'rb')):
         counts[filename + '-IN' ][line[1]] += 1
         counts[filename + '-OUT'][line[2]] += 1

answered Oct 2, 2012 at 20:07

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

pb100 Over a year ago

Python 2.6.4 (r264:75706, Apr 2 2012, 20:24:27) [C] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> import collections >>> counts = collections.defaultdict(collections.Counter) Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'module' object has no attribute 'Counter'

pb100 Over a year ago

From stackoverflow.com/questions/5079790/… : dictionary = collections.defaultdict(lambda: collections.defaultdict(int)) What is a difference between these two definitions ?

pb100 Over a year ago

And this docs.python.org/library/collections.html says that defaultdict is available since 2.5

Martijn Pieters Over a year ago

@przemol: A Counter offers more functionality than a defaultdict with an int value, such as retrieving the top counts, and combining multiple counters in various ways. Read the linked documentation for more details.

pb100 Over a year ago

which version of python should I have to be able to use collections.Counter ?

|

Tooblippe · Accepted Answer · 2012-10-03 18:50:52Z

1

I have recently started using the Pandas data frame. It has a CSV reader and makes slicing and dicing data very simple.

answered Oct 3, 2012 at 18:50

Tooblippe

3,7313 gold badges20 silver badges25 bronze badges

Collectives™ on Stack Overflow

python - how to count number of numbers from columns?

2 Answers 2

6 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related