Append multiple header information fields to file until next header found

Question

I am attempting to parse many log files using bash. The log files look something like this:

"",1/8/2016
"Timezone",-6

"Serial No.","000001"
"Location:","LS_trap_2c"
"High temperature limit (�C)",-20
"Low temperature limit (�C)",-40
"Date - Time","Temperature (�C)"
"8/11/2015 12:00",28.0
"8/11/2015 14:00",28.5
"8/11/2015 16:00",24.0

"",1/8/2016
"Timezone",-6

"Serial No.","000002"
"Location:","LS_trap_2D"
"High temperature limit (�C)",-20
"Low temperature limit (�C)",-40
"Date - Time","Temperature (�C)"
"8/11/2015 12:00",28.0
"8/11/2015 14:00",28.5

I want to append the Serial no., and Location (and maybe others later on) onto each line until the next header is reached, and output this to a master.csv file. The file should look like this ultimately:

"",1/8/2016
"Timezone",-6

"Serial No.","000001"
"Location:","Trap_2c"
"High temperature limit (�C)",-20
"Low temperature limit (�C)",-40
"Date - Time","Temperature (�C)"
LS_trap_2c,000001,"8/11/2015 12:00",28.0
LS_trap_2c,000001,"8/11/2015 14:00",28.5
LS_trap_2c,000001,"8/11/2015 16:00",24.0

"",1/8/2016
"Timezone",-6

"Serial No.","00002"
"Location:","LS_trap_2D"
"High temperature limit (�C)",-20
"Low temperature limit (�C)",-40
"Date - Time","Temperature (�C)"
LS_trap_2D,00002,"8/11/2015 12:00",28.0
LS_trap_2D,00002,"8/11/2015 14:00",28.5

Here is a question that helped me do processing on similar files using bash sed:

Bash Append header information to each line of a file until next header found

This oneliner is great for finding a header, storing it in holdspace and adding it into the front of each line

sed -r '/^"/h;//!{G;s/(.*)\n.*"(.*)"/\2,\1/}' fil.csv  >masfil.csv

This approach has not worked for appending multiple strings to the front since I am unsure of how to use multiple hold spaces with sed. Also, I'm not sure if sed is the best way to do this. I'm not very familiar with sed so any pointers would be greatly appreciated.

karakfa · Accepted Answer · 2017-02-15 04:06:28Z

1

awk to the rescue!

assuming your data is consistent

awk -F, '/"Serial No."/ {sn = $2} 
         /"Location:"/  {loc = $2} 
         /"([0-9]{1,2}\/){2}[0-9]{4} [0-9]{2}:[0-9]{2}"/ 
                        {$0 = loc FS sn FS $0}1' file

you can get rid of the quotes as well with gsub(/"/,"",$2) when assigning sn and loc, but not sure to remove them since the rest of the fields are quoted.

answered Feb 15, 2017 at 4:06

karakfa

67.8k8 gold badges45 silver badges59 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

5r9n Over a year ago

Thanks for the quick answer- evidently my data is not consistent- your awk works with the data I posted, but not with my real data (I thought they were identical). The serial number lengths are different and the location names are varied, but still...it only works with the data posted...Can you point toward a resource for understanding the awk you gave me?

5r9n Over a year ago

Your awk works, but only when I copy/paste the data into a new file. I'm thinking this is possibly due to one/multiple carriage return(s), but I'm not sure.

5r9n Over a year ago

That exactly what I did : stackoverflow.com/questions/42242852/…

Benjamin W. · Accepted Answer · 2017-02-15 04:56:42Z

There is only one hold space in sed, but you don't need multiple in this case:

/^"Serial No."/ {       # If we are on the "Serial No." line...
    N                   # Append next line to pattern space
    h                   # Copy pattern space to hold space
    # Remove everything but location and serial number from pattern space
    s/"[^"]*","([^"]*)"\n"[^"]*","([^"]*)"/\1,\2,/
    x                   # Swap pattern space and hold space
}
/^"[[:digit:]]/ {       # We are on a line where we want to prepend our data
    G                   # Append hold space to pattern space
    s/(.*)\n(.*)/\2\1/  # Move hold space content to front of pattern space
}

If this is stored in a file sedscr.sed, it can be called like

sed -E -f sedscr.sed infile

This does remove the double quotes as show in the example input/output; it also assumes that the lines where the data should be prepended are lines with a date, i.e., starting with a double quote and a digit.

Collectives™ on Stack Overflow

Append multiple header information fields to file until next header found

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related