How to parse log file with different types of messages

Question

I have a log file which contains complicated message types. Here is an example:

2016-07-07 13:30:02 [Main] *** Program start ***
2016-07-07 13:30:02 [UnzipFile] Before file collection
2016-07-07 13:30:02 [GetZipCol] Start get sorted zip file collection
2016-07-07 13:30:02 [GetZipCol] End get sorted zip file collection
2016-07-07 13:30:02 [Main] [ERROR] No unzip file
2016-07-07 13:30:03 [Main] *** Program end ***

The following grok pattern is only suitable for first 4 lines but not the 5th line.

grok{
    match => {"message" => ['%{Date:Date}%{SPACE}%{Time:Time}%{SPACE}%{WORD:Job}%{SPACE}%{GREEDYDATA:Message}']}
        }

I would like to know how should I modify the grok pattern as to capture[ERROR] from the last message. Is there anyone know how the way to do this?

This is my output part in conf

if [Message] == "*** Program start ***" {
    elasticsearch { 
    hosts => ["localhost:9200"] 
    index => "log-%{+YYYY.MM.dd}"
    template => "C:/logstash/log.json"
    template_overwrite => true
    }   
}
if [Message] == "*** Program end ***" {
    elasticsearch { 
    hosts => ["localhost:9200"] 
    index => "log-%{+YYYY.MM.dd}"
    template => "C:/logstash/log.json"
    template_overwrite => true
    }   
}  
if [Level] =~ /.+/ {
    elasticsearch { 
    hosts => ["localhost:9200"] 
    index => "log-%{+YYYY.MM.dd}"
    template => "C:/logstash/log.json"
    template_overwrite => true
    }   
}

If I only want to grasp the event when the Program starts and ends and also the events with errors while the other events can be dropped. However, according to what I have written. I can only grasp the data with [Error]. How should I also grasp the other data? And will there be a simpler way of doing that instead of typing 3 if conditional statements? Thanks.

Thanks.

"I can only grasp the data with [Error]" That's not true, the last conditional statement means that you grasp all messages which are successfully parsed by the pattern with the fiel Level — baudsp
– baudsp, Commented Jul 8, 2016 at 13:43
Instead of doing three conditional, you can do one, negating the three different conditions and dropping all the messages that pass => if ([Message] != "*** Program start ***" and [Message] != "*** Program end ***" and [Level] !~ /.+/) { drop{} }. And then you use one elasticsearch output — baudsp
– baudsp, Commented Jul 8, 2016 at 13:45
@baudsp Thanks, it works well. Just want to know how can I check if the Message contains the word "Program" instead of the whole line of message? — Kennedy Kan
– Kennedy Kan, Commented Jul 11, 2016 at 3:15
Like what you did with [Level], you can use the =~ comparator like this: =~ /Program/ — baudsp
– baudsp, Commented Jul 11, 2016 at 7:36

baudsp · Accepted Answer · 2018-04-27 14:50:04Z

1

You can use two pattern in the same grok filter, if the first one fails, the second is use. So in your case, the first pattern will try to capture the [ERROR], the second will be the pattern from your answer.
I think it's more readable.

grok{
  match => {
    "message" => [
       '%{DATE:Date}%{SPACE}%{TIME:Time}%{SPACE}\[%{WORD:Job}\]%{SPACE}\[%{WORD:Level}\]%{SPACE}%{GREEDYDATA:Message}',
       '%{DATE:Date}%{SPACE}%{TIME:Time}%{SPACE}\[%{WORD:Job}\]%{SPACE}%{GREEDYDATA:Message}'
   ]}
}

edited Apr 27, 2018 at 14:50

answered Jul 7, 2016 at 9:32

baudsp

4,1091 gold badge20 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Kennedy Kan Over a year ago

I have managed to follow your links to do some filtering, however, I cannot get the expected result. I have edited my question so as to be clear of my issues. Thanks if there is any help.

baudsp Over a year ago

@KennedyKan What is the expected result?

Kennedy Kan Over a year ago

I have solved the issue by refering to your comments to my question. Thanks so much

user84592 Over a year ago

Does there misssing a comma ',' between two regex, something like 'regex1',\n'regex2'

baudsp Over a year ago

@user84592 Good catch. Thank you.

jaap3 · Accepted Answer · 2016-07-07 07:51:45Z

0

I'm no expert on logstash, but from a quick glance at the docs it seems that these "grok" patterns are an abstraction on top of plain regular expressions.

So adding an optional non capturing group for the ERROR level message might work. i.e. (?:\[%{WORD:Level}\]%{SPACE})?. So that would make the full line read:

grok{
    match => {"message" => ['%{Date:Date}%{SPACE}%{Time:Time}%{SPACE}%{WORD:Job}(?:\[%{WORD:Level}\]%{SPACE})?%{GREEDYDATA:Message}']}
}

For reference I used this part of the documentation: https://www.elastic.co/guide/en/logstash/current/config-examples.html#_processing_syslog_messages

answered Jul 7, 2016 at 7:51

jaap3

2,8461 gold badge22 silver badges36 bronze badges

3 Comments

Kennedy Kan Over a year ago

Thanks for the answer. Just want to know how can I drop any events that do not contain Error? Thanks.

baudsp Over a year ago

Cf this answer: stackoverflow.com/a/38072094/6113627 Or all the other anwser : stackoverflow.com/search?q=%5Blogstash%5Ddrop

Kennedy Kan Over a year ago

I have managed to follow your links to do some filtering, however, I cannot get the expected result. I have edited my question so as to be clear of my issues. Thanks if there is any help.

Collectives™ on Stack Overflow

How to parse log file with different types of messages

2 Answers 2

5 Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related