How to catch duplicate entries in text file in linux [duplicate]

Question

Text file:

I want to catch 1 1 as duplicated

Lev Levitsky · Accepted Answer · 2015-08-09 18:06:41Z

121

Your question is not quite clear, but you can filter out duplicate lines with uniq:

sort file.txt | uniq

or simply

sort -u file.txt

(thanks RobEarl)

You can also print only repeating lines with

sort file.txt | uniq -d

answered Oct 5, 2012 at 9:57

Lev Levitsky

66.4k23 gold badges155 silver badges184 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

sort -u file.txt would also filter out duplicates

actually that works for me sort file.txt | uniq -d Thank, Lev

I will keep in mind thank you, Lev.

Steve · Accepted Answer · 2012-10-05 11:39:13Z

5

One way using GNU awk:

awk 'array[$0]++' file.txt

Results:

1 1

answered Oct 5, 2012 at 11:39

Steve

55.1k13 gold badges94 silver badges105 bronze badges

MLSC · Accepted Answer · 2014-07-21 19:13:58Z

1

You can use it easily:

sort -u file.txt

OR

awk '!x[$0]++' file.txt

answered Jul 21, 2014 at 19:13

MLSC

5,9728 gold badges60 silver badges90 bronze badges