Logstash grok parse error parsing log file

Question

I am trying to parse this log format:

http://localhost:8080/,200,OK,11382,date=Mon  27 Apr 2015 12:56:33 GMT;newheader=foo;connection=close;content-type=text/html;charset=ISO-8859-1;server=Apache-Coyote/1.1;

with this config file:

input {
  stdin{}
}


filter {
  grok {
        match => [ "message" , "%{URI:uriaccessed},%{NUMBER:httpcode},%{WORD:httpcodeverb},%{NUMBER:bytes},date=%{TIMESTAMP_ISO8601:logtimestamp};%{GREEDYDATA:msg}"]   
  }
  mutate{
        convert => ["httpcode","integer"]
        convert => ["bytes","integer"]
  }


  date {
    locale => "en"
    match => [ "logtimestamp" , "EEE dd MMM yyy HH:mm:ss" ] #Mon  27 Apr 2015 12:56:33 GMT
  }
}


output {
  stdout { codec => rubydebug }
}

However, I am getting a grok prase failure, I am not sure what the problem is. cant seem to pin point the pattern that is causing the problem. Any thoughts/comments would be appreciated.

Alain Collins · Accepted Answer · 2015-05-01 15:23:38Z

3

TIMESTAMP_ISO8601 matches:

%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?

and your date is not in that format. There doesn't seem to be a predefined pattern for you, so here's one that will work:

%{DAY} +%{MONTHDAY} %{MONTH} %{YEAR} +%{TIME} %{WORD}

Note that %{TZ} doesn't like GMT, so I used %{WORD}.

Good luck.

answered May 1, 2015 at 15:23

Alain Collins

16.4k2 gold badges34 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

user3498894 · Accepted Answer · 2015-05-01 16:26:40Z

1

The timestamp in your log example does not match the TIMESTAMP_ISO8601 pattern. You could try other patterns, but I suspect the double space between "Mon" and "27" will be a problem, and I don't see "GMT" matching the TZ pattern. You could try adding your own pattern file with a TZORGMT entry and then use it in a match like this:

TZORGMT (?:[PMCE][SD]T|UTC|GMT)
%{URI:uriaccessed},%{NUMBER:httpcode},%{WORD:httpcodeverb},%{NUMBER:bytes},date=%{DAY}  %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} %{TZORGMT}

The grok debugger at https://grokdebug.herokuapp.com/ is helpful for debugging things like this.

If you're then going to do a date { match }, you'll need to write a pattern for that, something like:

"dd MM YYYY HH:mm:ss ZZ"

answered May 1, 2015 at 16:26

user3498894

111 bronze badge

1 Comment

Yogesh_D Over a year ago

Thanks! For the grokdebug link, I had checked it out earlier, and was always trying put in the patterns from the logstash patterns file in there as custom patterns. This time when I revisited the site, I realized that it does have all the logstash patterns build in. Thanks!

Collectives™ on Stack Overflow

Logstash grok parse error parsing log file

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related