Bash/Shell : CSV File Processing Issue

Question

I have written a bash script that calls a Python script to scan for a vulnerability:

#scan.sh
while IFS=, read -r n d ;do
   python scan.py $d 443 | sed "s/^\(.\)/$n,\1/"
done < Input.csv

Input.csv looks like:

#Input.csv
1,abc.com
2,xyz.com
3,pqr.com
.........

Output.csv should look like:

1,abc.com,True,2016-03-06
1,abc.com,False,2016-03-06
1,abc.com,True,2016-03-06
1,abc.com,True,2016-03-06
2,xyz.com,False,2016-03-06
2,xyz.com,False,2016-03-06
2,xyz.com,False,2016-03-06
2,xyz.com,False,2016-03-06
3,pqr.com,True,2016-03-06
3,pqr.com,True,2016-03-06
3,pqr.com,True,2016-03-06
3,pqr.com,True,2016-03-06
........................

Each ID will have four entries in the output.csv with varying results.

If any of the four entries for an ID is True, it has to be True and need to have one entry for a ID.
If all four entries for an ID are False, it has to be False and with one entry.
If all four entries for an ID are True, it has to be True and with one entry.

Thus:

# processed_out.csv

1,abc.com,True,2016-03-06
2,xyz.com,False,2016-03-06
3,pqr.com,True,2016-03-06
........

How can I achieve this case only by using awk/sed?

FYI : I'm a beginner in bash scripting.

any reason you want to do this in bash? You've already got a python script doing the "Important" work. wouldn't it be easier to just have that script spit out the data in your desired format in the first place? — Marc B
– Marc B, Commented Mar 23, 2016 at 18:33
@MarcB I don't want alter any single line edit in the master script and to be frank i'm more interested in learning something new in bash programming ;) — Arun
– Arun, Commented Mar 23, 2016 at 18:36
@MarcB Even in the #scan.sh script, I'm inserting/prefixing the ID of each domain ( sed "s/^(.)/$n,\1/" ) to the output result. I'm in love with bash actually ! — Arun
– Arun, Commented Mar 23, 2016 at 18:46

glenn jackman · Accepted Answer · 2016-03-23 18:50:33Z

3

It's pretty straightforward with awk:

awk -F, -v OFS=, '
    $3 == "True" {ntrue++} 
    NR%4 == 0 {
        $3 = (ntrue > 0) ? "True" : "False"
        print
        ntrue = 0
    }
' output.csv > processed_out.csv

answered Mar 23, 2016 at 18:50

glenn jackman

249k42 gold badges233 silver badges362 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Arun Over a year ago

Thank you ! It's clean method

karakfa · Accepted Answer · 2016-03-23 18:55:43Z

2

awk to the rescue!

$ pr -4ats, file | 
  awk -F, -v OFS=, '{for(i=3;i<=NF;i+=4) 
                        if($i=="True") {$3=$i; break}; 
                     print $1,$2,$3,$4}'

1,abc.com,True,2016-03-06
2,xyz.com,False,2016-03-06
3,pqr.com,True,2016-03-06

pr to bring the related records together, override status with any "True" value in the corresponding positions.

answered Mar 23, 2016 at 18:55

karakfa

67.8k8 gold badges45 silver badges59 bronze badges

1 Comment

glenn jackman Over a year ago

Nice. An alternative to pr is paste -d, - - - - < file

Collectives™ on Stack Overflow

Bash/Shell : CSV File Processing Issue

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related