Remove duplicate data using bash, sed or awk

Question

How can I search for duplicate data using batch, sed or awk? Goal is to remove duplicate "Changelist: XXXXX" entry from data.txt file. I'm kind of stuck, can somebody help me?

Please take a look at output.txt for desired output.

data.txt

====================================
 Changelist: 808298
 Date: 2015/03/19
 Developer: A
 ShortDescr: Checking in the following graphics:

 CodeReview: 
 CodeReview: Result: @result___
 ====================================
 Changelist: 808273
 Date: 2015/03/19
 Developer: B
 ShortDescr: Hello

 CodeReview: Result: 
 ====================================
 Changelist: 808271
 Date: 2015/03/19
 Developer: C
 ShortDescr: HI

 CodeReview: 
 ====================================
 Changelist: 808298
 Date: 2015/03/19
 Developer: A
 ShortDescr: Checking in the following graphics:

 CodeReview: 
 CodeReview: Result: @result___
 ====================================
 Changelist: 808273
 Date: 2015/03/19
 Developer: B
 ShortDescr: Hello

 CodeReview: Result:  
 ====================================
  Changelist: 808277
 Date: 2015/03/19
 Developer: D
 ShortDescr: HEY

 CodeReview: 
 ====================================

output.txt

    ====================================
     Changelist: 808298
     Date: 2015/03/19
     Developer: A
     ShortDescr: Checking in the following graphics:

     CodeReview: 
     CodeReview: Result: @result___
     ====================================
     Changelist: 808273
     Date: 2015/03/19
     Developer: B
     ShortDescr: Hello

     CodeReview: Result: 
     ====================================
     Changelist: 808271
     Date: 2015/03/19
     Developer: C
     ShortDescr: HI

     CodeReview: 
     ====================================
      Changelist: 808277
     Date: 2015/03/19
     Developer: D
     ShortDescr: HEY

     CodeReview: 
     ====================================


glen's output.txt

 ====================================
 Changelist: 808298
 Date: 2015/03/19
 Developer: A
 ShortDescr: Checking in the following graphics:

 CodeReview:
 ====================================
 Changelist: 808273
 Date: 2015/03/19
 Developer: B
 ShortDescr: Hello

 CodeReview:
 ====================================
 Changelist: 808271
 Date: 2015/03/19
 Developer: C
 ShortDescr: HI

 CodeReview: 
 ====================================
  Changelist: 808277
 Date: 2015/03/19
 Developer: D
 ShortDescr: HEY

 CodeReview: 
 ====================================
 Changelist: 808298
 Date: 2015/03/19
 Developer: A
 ShortDescr: Checking in the following graphics:

 CodeReview:
 ====================================$sep

are you sure the data sharing the same ChangeList will be extactly the same? — Jason Hu
– Jason Hu, Commented Mar 20, 2015 at 16:29

glenn jackman · Accepted Answer · 2015-03-20 16:32:53Z

2

This is actually a very common task with awk

sep='====================================\n'
awk -F'\n' -v RS="$sep" -v ORS="$sep" '!seen[$1]++' data.txt > output.txt

Here, we're using the $sep as the awk record separator to read the paragraphs, and newline as the field separator

!seen[$1]++ is an expression that is only true for the first record where this particular field 1 is encountered. Since no action is given, the default action is to print the current record, with the output record separator appended.

answered Mar 20, 2015 at 16:32

glenn jackman

249k42 gold badges233 silver badges362 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Mihir Over a year ago

glenn thanks for quick response. but it didn't remove that duplicate block.

glenn jackman Over a year ago

hmm, did for me. What's your OS?

Mihir Over a year ago

i'm running on windows

Jason Hu Over a year ago

it seems your environment didn't perform extension. are you running some kind of shell?

glenn jackman Over a year ago

I think that single quotes have no special meaning, so try changing all single quotes to double quotes

|

Collectives™ on Stack Overflow

Remove duplicate data using bash, sed or awk

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related