Removing Duplicates from CSV in Shell

Question

I am looking for help in below file. There are clients in column 1 which can be part of one or multiple groups. Their status can be either Failed, succeeded or interrupted. I want only those client who is not having any entry of succeeded.

Example

My file is as below

RBCSREXC04 AUTO_RERUN_RBC_DAILY succeeded
RBCSRTM03 AUTO_RERUN_RBC_DAILY succeeded
RBCVMAPPPROD01 AUTO_RERUN_RBC_DAILY succeeded
RBCVVMAPPDEV02 AUTO_RERUN_RBC_DAILY succeeded
E6-RBC-SQL-06 AUTO_RERUN_RBC_DAILY succeeded
E6-ODI-Prod-01 AUTO_RERUN_RBC_DAILY succeeded
GSIERBC2004 AUTO_RERUN_RBC_DAILY succeeded
GSIERBC3008 AUTO_RERUN_RBC_DAILY succeeded 
RBCSRTM03 D_RBC_VM_DUBLIN_E6 failed
RBCSREXC04 D_RBC_VM_DUBLIN_E6 failed
GSIERBC3008 D_RBC_VM_DUBLIN_E6_1 interrupted
E6-ODI-Prod-01 D_RBC_VM_DUBLIN_E6_1 failed
RBCVVMAPPDEV02 D_RBC_VM_DUBLIN_E6_1 failed
E6-RBC-SQL-06 D_RBC_VM_DUBLIN_E6 failed
RBCVMAPPPROD01 D_RBC_VM_DUBLIN_E6 failed
RBCSRCV01 D_RBC_VM_DUBLIN_E6 failed

Below is the Expected Output

RBCSRCV01 D_RBC_VM_DUBLIN_E6 failed

Why isn't b group2 failed in your output? Or others? Is it just one random matching line per group? — Shawn
– Shawn, Commented Apr 10, 2020 at 12:37

Freddy · Accepted Answer · 2020-04-14 10:51:48Z

1

You could maintain two arrays with awk for the "good" and the "bad" entries where the array index is the first column and only print the "bad" ones for which no entry in the "good" array exists.

awk '
  $3=="succeeded"{ good[$1] }  # we only need the index here
  $3=="failed" || $3=="interrupted"{
    if ($1 in bad){ 
      bad[$1]=bad[$1] ORS $0 # append this line to existing entry
    } else {
      bad[$1]=$0             # save the line
    }
  }
  END{
    for (i in bad)
      if (!(i in good))print bad[i]
  }
' file

edited Apr 14, 2020 at 10:51

answered Apr 10, 2020 at 13:47

Freddy

4,7581 gold badge9 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

GURU SINGH Over a year ago

Thanks, Freddy for your help but in your script, I am facing issue like if the client gets succeeded in other groups then it shows it as Failed. But I want that if a client gets successful in the same or any other group, it should not come in output. I want only those clients who have single or multiple entries of Failed or interrupted with no success.

GURU SINGH Over a year ago

Freddy, I have updated my question with a new example. I want only those clients in output who are not at all success even once in any of the group.

GURU SINGH Over a year ago

Thanks, @Freddyfor your help but I am facing the same issue, if the client gets succeeded in any other group then it does not filter it but if it gets succeeded in the same group then it does. I want if the client gets succeeded in any of the group then it should not be in output –

GURU SINGH Over a year ago

I have updated my question with the original file from my server.

Freddy Over a year ago

I replaced success with succeeded in my script.

|

uzsolt · Accepted Answer · 2020-04-14 11:10:52Z

1

sed -n '/succeeded/ s,\(.*\) .* succeeded$,\1,p' status.csv | grep -v -f - status.csv

Explanation: the sed produces list of 'x' which has success status (in any group). The grep will drop (because of -v) them from status.csv (the -f - means grep obtains patterns from stdin (equals sed's stdout)).

edited Apr 14, 2020 at 11:10

answered Apr 10, 2020 at 13:55

uzsolt

6,0872 gold badges24 silver badges35 bronze badges

5 Comments

GURU SINGH Over a year ago

Thanks, UZSOLT for your help but in your script, I am facing issue like if the client gets succeeded in other groups then it shows it as Failed. But I want that if a client gets successful in the same or any other group, it should not come in output. I want only those clients who have single or multiple entries of Failed or interrupted with no success.

GURU SINGH Over a year ago

Thanks, @Uzsolt for your help but I am facing the same issue, if the client gets succeeded in any other group then it does not filter it but if it gets succeeded in same group then it does. I want if the client gets succedded in any of the group then it should not be in output

uzsolt Over a year ago

@GURUSINGH success -> succeeded Please don't edit many times your question because it's impossible answer it. Please be exact.

GURU SINGH Over a year ago

Thanks @uzsolt. I am sorted now. Please let me know if we can do achieve the same thing in Powershell also

uzsolt Over a year ago

I don't know Powershell. Maybe it should be an another question.

Collectives™ on Stack Overflow

Removing Duplicates from CSV in Shell

2 Answers 2

9 Comments

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

9 Comments

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related