Adding another column to awk output

Question

I have a HAProxy log file with content similar to this:

Feb 28 11:16:10 localhost haproxy[20072]: 88.88.88.88:6152 [28/Feb/2017:11:16:01.220] frontend backend_srvs/srv1 9063/0/0/39/9102 200 694 - - --VN 9984/5492/191/44/0 0/0 {Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|http://subdomain.domain.com/location1} "GET /location1 HTTP/1.1"
Feb 28 11:16:10 localhost haproxy[20072]: 88.88.88.88:6152 [28/Feb/2017:11:16:10.322] frontend backend_srvs/srv1 513/0/0/124/637 200 14381 - - --VN 9970/5491/223/55/0 0/0 {Mozilla/5.0 AppleWebKit/537.36 Chrome/56.0.2924.87 Safari/537.36|http://subdomain.domain.com/location2} "GET /location2 HTTP/1.1"
Feb 28 11:16:13 localhost haproxy[20072]: 88.88.88.88:6152 [28/Feb/2017:11:16:10.960] frontend backend_srvs/srv1 2245/0/0/3/2248 200 7448 - - --VN 9998/5522/263/54/0 0/0 {another user agent with fewer columns|http://subdomain.domain.com/location3} "GET /location3 HTTP/1.1"
Feb 28 11:16:13 localhost haproxy[20072]: 88.88.88.88:6152 [28/Feb/2017:11:16:10.960] frontend backend_srvs/srv1 2245/0/0/3/2248 200 7448 - - --VN 9998/5522/263/54/0 0/0 {Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36|} "GET /another_location HTTP/1.1"

I want to extract some of the fields in order to have the following output:

 Field 1             Field 2           Field 3         Field 4         Field 5         Field 6
Date/time       HTTP status code     HTTP Method       Request      HTTP version    Referer URL

Basically, in this particular case the output should be:

Feb 28 11:16:10  200 GET /location1 HTTP/1.1    http://subdomain.domain.com/location1
Feb 28 11:16:10  200 GET /location2 HTTP/1.1    http://subdomain.domain.com/location2
Feb 28 11:16:13  200 GET /location3 HTTP/1.1    http://subdomain.domain.com/location3
Feb 28 11:16:13  200 GET /another_location HTTP/1.1

The only problem here is extracting the Referer URL which is between curly brackets together with the user agent and they're separated by a pipe. Also, the user agent has a variable number of fields.

The only solution I could think of was extracting the referer url separately and then pasting the columns together:

requests_temp=`grep -F " 88.88.88.88:" /root/file.log | tr -d '"'`
requests=`echo "${requests_temp}" | awk '{print $1" "$2" "$3"  "$11, $(NF-2), $(NF-1), $NF}' > /tmp/requests_tmp`
referer_url=`echo "${requests_temp}" | awk 'NR > 1 {print $1}' RS='{' FS='}' | awk -F'|' '{ print $2 }' > /tmp/referer_url_tmp`

paste /tmp/abuse_requests_tmp /tmp/referer_url_tmp

But I don't really like this method. Is there any other way in which I can do it using only one awk line? Maybe assign the referer url column to a variable inside awk and then using it to create the same output?

VIPIN KUMAR · Accepted Answer · 2017-02-28 11:59:27Z

1

try below solution -

awk '/88.88.88.88/ {gsub(/"/,"",$0);split($(NF-3),a,"|"); {print $1,$2,$3,$11, $(NF-2), $(NF-1), $NF, substr(a[2],1,(length(a[2])-1))}}' a
Feb 28 11:16:10 200 GET /location1 HTTP/1.1 http://subdomain.domain.com/location1
Feb 28 11:16:10 200 GET /location2 HTTP/1.1 http://subdomain.domain.com/location2
Feb 28 11:16:13 200 GET /location3 HTTP/1.1 http://subdomain.domain.com/location3
Feb 28 11:16:13 200 GET /another_location HTTP/1.1

answered Feb 28, 2017 at 11:59

VIPIN KUMAR

3,1572 gold badges25 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

oliv · Accepted Answer · 2017-02-28 12:11:28Z

1

You can do all at once using awk:

awk '$6 ~ /88\.88\.88\.88:[0-9]+/{
   split($0,a,/[{}]/)
   $0=a[1] OFS a[3]
   split(a[2],b,"|")
   print $1,$2,$3,$11,substr($18,2),$19,substr($20,1,length($20)-1),b[2]
}' file.log

The first split is splitting the variable part of line (included in between the {...}) into the array a.

The line is rebuilt in order to have a fix number of fields $0=a[1] OFS a[3]

The second split allows extracting the URL from variable based on | characters.

At last the print shows all needed elements. Note the substr are here for removing the ".

edited Feb 28, 2017 at 12:11

answered Feb 28, 2017 at 11:59

oliv

13.3k30 silver badges52 bronze badges

1 Comment

VIPIN KUMAR Over a year ago

You forgot to add the filter for ip address (88.88.88.88), if my file has one more value with different ip, that also will get print with your solution.

Collectives™ on Stack Overflow

Adding another column to awk output

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related