Grok Pattern to Extract Syslog in Linux(Ubuntu)

Question

If you have any grok pattern to extract syslog in ubuntu please provide it. Thank You!

Edited --->>

My syslog example ->

"Aug 20 15:53:02 amantha-ubuntu-server kibana[1877]: {\"type\":\"response\",\"@timestamp\":\"2020-08-20T10:23:02Z\",\"tags\":[],\"pid\":1877,\"method\":\"post\",\"statusCode\":200,\"req\":{\"url\":\"/internal/search/es\",\"method\":\"post\",\"headers\":{\"connection\":\"upgrade\",\"host\":\"example.com\",\"content-length\":\"861\",\"kbn-version\":\"7.8.1\",\"user-agent\":\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36\",\"content-type\":\"application/json\",\"accept\":\"*/*\",\"origin\":\"http://example.com\",\"referer\":\"http://example.com/app/kibana\",\"accept-encoding\":\"gzip, deflate\",\"accept-language\":\"en-US,en;q=0.9,si;q=0.8\"},\"remoteAddress\":\"127.0.0.1\",\"userAgent\":\"127.0.0.1\",\"referer\":\"http://example.com/app/kibana\"},\"res\":{\"statusCode\":200,\"responseTime\":65,\"contentLength\":9},\"message\":\"POST /internal/search/es 200 65ms - 9.0B\"}"

I tried following filter ->

filter {
      grok {
            match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"}
            #match => {"syslog_message" => "%{WORD:FILTERED}" }
            #add_field => [ "received_at", "%{@timestamp}" ]
            #add_field => [ "received_from", "%{host}" ]
            remove_field => ["host","message"]
            
      }

      mutate{

            rename => ["@timestamp","time"]

      }
}

Then I got the below output. And I want to extract the syslog message part.

 "time" => 2020-08-20T11:17:57.995Z,
      "syslog_message" => "message repeated 9 times: [ {\"type\":\"response\",\"@timestamp\":\"2020-08-20T10:27:22Z\",\"tags\":[],\"pid\":1877,\"method\":\"get\",\"statusCode\":200,\"req\":{\"url\":\"/api/rollup/indices\",\"method\":\"get\",\"headers\":{\"connection\":\"upgrade\",\"host\":\"example.com\",\"kbn-version\":\"7.8.1\",\"user-agent\":\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36\",\"content-type\":\"application/json\",\"accept\":\"*/*\",\"referer\":\"http://example.com/app/kibana\",\"accept-encoding\":\"gzip, deflate\",\"accept-language\":\"en-US,en;q=0.9,si;q=0.8\"},\"remoteAddress\":\"127.0.0.1\",\"userAgent\":\"127.0.0.1\",\"referer\":\"http://example.com/app/kibana\"},\"res\":{\"statusCode\":200,\"responseTime\":31,\"contentLength\":9},\"message\":\"GET /api/rollup/indices 200 31ms - 9.0B\"},]",
    "syslog_timestamp" => "Aug 20 15:58:52",
                "path" => "/var/log/syslog",
            "@version" => "1",
     "syslog_hostname" => "amantha-ubuntu-server",
      "syslog_program" => "amantha"

give as some lines from your syslog as example

roodyCool
– roodyCool

2020-08-20 10:05:01 +00:00
Commented Aug 20, 2020 at 10:05 — roodyCool
– roodyCool, Commented Aug 20, 2020 at 10:05
Added in the question!

Amantha
– Amantha

2020-08-20 11:33:39 +00:00
Commented Aug 20, 2020 at 11:33 — Amantha
– Amantha, Commented Aug 20, 2020 at 11:33

aaaa · Accepted Answer · 2020-08-20 13:59:47Z

1

The first thing I'd do is use gsub on that line to remove the " and \.

You might use:

mutate {
 gsub => [ "message", "[\\\"]", "" ]
}

This'll leave you with:

Aug 20 15:53:02 amantha-ubuntu-server kibana[1877]: {type:response,@timestamp:2020-08-20T10:23:02Z,tags:[],pid:1877,method:post,statusCode:200,req:{url:/internal/search/es,method:post,headers:{connection:upgrade,host:example.com,content-length:861,kbn-version:7.8.1,user-agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36,content-type:application/json,accept:*/*,origin:http://example.com,referer:http://example.com/app/kibana,accept-encoding:gzip, deflate,accept-language:en-US,en;q=0.9,si;q=0.8},remoteAddress:127.0.0.1,userAgent:127.0.0.1,referer:http://example.com/app/kibana},res:{statusCode:200,responseTime:65,contentLength:9},message:POST /internal/search/es 200 65ms - 9.0B}

And you can use the following line to grab the "syslog message" after gsub. I don't really know if you wanted it broken down any further, but more than happy to help if that's desired.

(?<syslog_timestamp>%{SYSLOGTIMESTAMP}) (?<syslog_hostname>%{SYSLOGHOST}) (?<syslog_program>%{SYSLOGPROG}): {(?<syslog_message>(?<={).*(?=}))}

answered Aug 20, 2020 at 13:59

aaaa

2461 silver badge9 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Amantha Over a year ago

Thank you very much. I want to brake down the rest of the message to get the browset type, Client ip etc... can you help me? Can you provide a filter to brake down the whole syslog.

aaaa Over a year ago

Yeah, I'll try to have a go at it soon. Need to crash at the moment, but I'll reply as soon as possible.

Amantha Over a year ago

Thank you very much.. I am stucked becuse of this problem.. If you provide the filter, it will be a great help.

aaaa Over a year ago

OK, here you go: pastebin.com/raw/2H2b5pVW This should break things down. Please let me know if that helps you.

Amantha Over a year ago

I will inform you. Thank you. Give me some time

|

Collectives™ on Stack Overflow

Grok Pattern to Extract Syslog in Linux(Ubuntu)

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related