Counting duplicates in a specific column using perl/linux

Question

I have a file(tab delimited) with 6 columns (here I have shown 2 columns for simplicity)

46_#1   A   
47_#1   B   
49_#1   C   
51_#1   D   
51_#1   E

I want to count duplicates in first column (only count-no removal) and store count in next column. So output should be-

46_#1   1  A    
47_#1   1  B    
49_#1   1  C    
51_#1   2  D    
51_#1   2  E

I have used linux command-

uniq -c  file

but this will take whole line (not 1st column) then I used

uniq -c -w5 file

But word count in first column can vary.

Can anyone help please?

PS- I have a very big file (around 1gb).

Oh!sorry I should have mention that. No they can be far away. — Vikas
– Vikas, Commented Jan 27, 2012 at 12:33

Borodin · Accepted Answer · 2012-01-27 12:27:43Z

6

I don't like just providing complete solutions, but it seemed the easiest way to explain. This program reads through the file twice: first to accumulate the frequency information and then to output the modified data.

use strict;
use warnings;

@ARGV or die "No input file specified";

open my $fh, '<', $ARGV[0] or die "Unable to open input file: $!";

my %count;

while (<$fh>) {
  next unless my ($key) = split;
  $count{$key}++;
}

seek $fh, 0, 0;
while (<$fh>) {
  chomp;
  next unless my ($key, $rest) = split ' ', $_, 2;
  print "$key $count{$key} $rest\n";
}

answered Jan 27, 2012 at 12:27

Borodin

127k9 gold badges72 silver badges146 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Vikas Over a year ago

Thanks a lot. It is perfectly fine.

Collectives™ on Stack Overflow

Counting duplicates in a specific column using perl/linux

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related