Is there way to delete duplicate header in a file in Unix?

Question

How can I delete multiple headers from a file? I tried to use the below code after finding it from How can I delete duplicate lines in a file in Unix?.

awk '!x[$0]++' file.txt

It is deleting all the duplicate records in the file. But in my case, I just need the header duplicates to be removed, not the duplicate records in the file. For example, I have a file with the below data:

column1, column2, column3, column4, column5
value11, value12, value13, value14, value14
value21, value22, value23, value24, value25
value31, value32, value33, value34, value35
value41, value42, value43, value44, value45
value51, value52, value53, value54, value55
value21, value22, value23, value24, value25
column1, column2, column3, column4, column5
value11, value12, value13, value14, value14
value21, value22, value23, value24, value25
column1, column2, column3, column4, column5
column1, column2, column3, column4, column5

I am expecting the output as below:

column1, column2, column3, column4, column5
value11, value12, value13, value14, value14
value21, value22, value23, value24, value25
value31, value32, value33, value34, value35
value41, value42, value43, value44, value45
value51, value52, value53, value54, value55
value21, value22, value23, value24, value25
value11, value12, value13, value14, value14
value21, value22, value23, value24, value25

tripleee · Accepted Answer · 2014-01-31 04:32:34Z

11

If you know that the first line contains the header, just delete all other instances of that.

awk 'FNR==1 { header = $0; print }
     $0 != header' file

If that won't work, please tell us how we can identify a header line. If it's just a static string, grep -vF 'that string' or if it matches a particular regex, grep -v 'that regex'.

edited Jan 31, 2014 at 4:32

answered Jan 30, 2014 at 17:28

tripleee

192k37 gold badges318 silver badges368 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Ed Morton Over a year ago

$0 == header { next }1 can be written as just $0 != header.

potong · Accepted Answer · 2014-01-31 07:16:52Z

2

This might work for you (GNU sed):

sed -r '1h;1!G;/^(.*)\n\1/d;P;D' file

answered Jan 31, 2014 at 7:16

potong

59.3k6 gold badges55 silver badges92 bronze badges

Collectives™ on Stack Overflow

Is there way to delete duplicate header in a file in Unix?

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related