I'm working with Elastic Stack 7.6.2.
Concatenate stack trace lines into one log entry
I'm sending logs to Logstash through Filebeat. I have to configure Filebeat so it treats the whole stack trace as one entry. I'm using multiline as described in Examples of multiline configuration:
#filebeat.yml
filebeat:
inputs:
- type: log
…
multiline:
pattern: '^[[:space:]]+(at|\.{3})[[:space:]]+\b|^Caused by:'
match: after
output:
logstash:
hosts: ["logstash:5044"]
Handle two types of log entries
In my logstash.conf file I have a filter matching against:
- a regular Spring Boot log entry (not covered here)
e.g.:
2020-05-12 08:31:26.530 INFO 10197 --- [SpringContextShutdownHook] o.s.s.c.ThreadPoolTaskExecutor : Shutting down ExecutorService 'applicationTaskExecutor'
java.lang.IllegalArgumentException: Exception message
at in.keepgrowing.springbootlog4j2scaffolding.SpringBootLog4j2ScaffoldingApplication.main(SpringBootLog4j2ScaffoldingApplication.java:14) [classes/:?]
at com.example.myproject.Author.getBookIds(Author.java:38)
at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Caused by: java.lang.NullPointerException
at com.example.myproject.Book.getId(Book.java:22)
at com.example.myproject.Author.getBookIds(Author.java:35)
... 1 more
Because I haven't listed multiple patterns in one match, every entry is being checked against both matches (I think the break_on_match is not working in this case). As a result the _grokparsefailure tag is added to all entries. To remove this tag I have to know that a particular entry was successfuly matched by one pattern - the stacktrace or spring_boot_log tag will be present in such a case. Therefore I can safely delete the _grokparsefailure tag for entries that have my tag:
# logstash.conf
…
filter {
grok {
match => { "message" => "%{JAVACLASS:exception}:\s%{GREEDYDATA:stacktrace}" }
add_tag => [ "stacktrace" ]
}
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp}…" }
add_tag => [ "spring_boot_log" ]
}
if "stacktrace" in [tags] or "spring_boot_log" in [tags] {
mutate {
remove_tag => ["_grokparsefailure"]
}
}
}
…
Below you can see the screenshot from my ElasticHQ showing how an example stack trace was parsed. There are two parts: exception and stacktrace, and my custom tag in the tags array:
Useful links:
(^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)pattern so that i can have new fields like Exception and causedBy so that i can later do a group by kinda thing. So can i put this pattern somewhere inline ? Please help.grok { match => { "message" => [ ... ] }}(e.g. you can supply a list of patterns inline). I'd try putting your exception pattern on before your regular pattern in the pattern list.