Removing columns in python

Question

Having a comma-separated file with around 50 columns and several rows, I need to remove all columns that are always 0 (i.e all values in that column are zero).

The file is read with the following piece of code:

with open('data.txt', 'rb') as f:
    reader.csv.reader(f, delimiter=',')
    for row in reader:
        print row


0 0.1 0.3 0.4 0
0 0.2 0.5 0.3 0
0 0.7 0.9 0.2 0

How one can exactly remove columns (that are 0) from this memory structure. It would be more better, if there is no re-writing and re-reading to another temporary csv file to achieve this.

How many rows are we talking about here? This is easiest to handle by reading it all into memory first. — Martijn Pieters
– Martijn Pieters, Commented Mar 25, 2013 at 15:13
Do you know ahead of time which columns are all zeros and need to be removed? Or do you have to determine that dynamically? — Travis Griggs
– Travis Griggs, Commented Mar 25, 2013 at 15:14

Martijn Pieters · Accepted Answer · 2013-03-25 16:02:18Z

1

Read in all rows (mapping all the values to floats), transform to columns using zip(*rows), only keep any that are have non-zero values using any(), transform back to rows using zip(*columns):

with open('data.txt', 'rb') as f:
    rows = list(map(float, row) for row in csv.reader(f, delimiter=','))

rows = zip(*[col for col in zip(*rows) if any(col)])

The latter step as a demonstration:

>>> rows = [[0, 0.1, 0.3, 0.4, 0], [0, 0.2, 0.5, 0.3, 0], [0, 0.7, 0.9, 0.2, 0]]
>>> zip(*[col for col in zip(*rows) if any(col)])
[(0.1, 0.3, 0.4), (0.2, 0.5, 0.3), (0.7, 0.9, 0.2)]

edited Mar 25, 2013 at 16:02

answered Mar 25, 2013 at 15:17

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Shahzad Over a year ago

Isn't this possible in python do it all in memory instead of writing to CSV file and re-reading it again. Because I need to process this data further.

Martijn Pieters Over a year ago

@Shahzad: This was all in memory. I thought you wanted it written to a CSV again; just remove the writing stage if you don't want that.

Collectives™ on Stack Overflow

Removing columns in python

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related