rsyslog variable from mmnormalize as part of omfile filename

Question

I have a log line that looks like this:

May 20 10:25:42 192.168.20.100 Timestamp="2024-05-20 10:25:42",LogId="535666280",NodeId="192.168.1.100",Facility="Packet Filtering",Type="Notification",Event="Connection discarded",Action="Discard",Protocol="1",Src="10.0.0.10",Dst="10.10.10.10",RuleId="21.0",Srcif="0",IcmpType="11",IcmpCode="0",CompId="Limes node 1",ReceptionTime="2024-05-20 10:25:42",SenderType="Firewall",SituationId="70019",Situation="Connection_Discarded",EventId="7198237666637962215",Service="Time Exceeded (TTL Exceeded)"^J

and I'd like to use the value of NodeId as part of the path to write the log to disk.

I've created a path template like so:

template (name="DynFile" type="list") {
  constant(value="/var/log/my/path/")
  property(name="$!customfromhost")
  constant(value=".log")
}

And I try to use mmnormalize to set the customfromhost variable:

ruleset(name="parsecsv") {
      action(type="mmnormalize" rule=["rule=:%logtime:date-rfc3164% %logsender:ipv4% Timestamp=%timestamp:quoted-string%,LogId=%logid:quoted-string%,NodeId=%gurkafromhost:quoted-string%,%rest:rest%"])
      action(type="omfile" dynaFile="DynFile")
}

I've defined input to use the ruleset like so:

input(type="imudp" port="514" ruleset="parsecsv")

The log ends up in /var/log/my/path/.log - so something is wrong with my variable - either the way I reference it or the way it is set.

I've tried verifying my rule by calling lognormalizer directly which seem to work:

$ cat ruleBase.rb 
rule=:%logtime:date-rfc3164% %logsender:ipv4% Timestamp=%timestamp:quoted-string%,LogId=%logid:quoted-string%,NodeId=%customfromhost:quoted-string%,%rest:rest%
$ cat line.log 
May 20 10:25:42 192.168.20.100 Timestamp="2024-05-20 10:25:42",LogId="535666280",NodeId="192.168.1.100",Facility="Packet Filtering",Type="Notification",Event="Connection discarded",Action="Discard",Protocol="1",Src="10.0.0.10",Dst="10.10.10.10",RuleId="21.0",Srcif="0",IcmpType="11",IcmpCode="0",CompId="Limes node 1",ReceptionTime="2024-05-20 10:25:42",SenderType="Firewall",SituationId="70019",Situation="Connection_Discarded",EventId="7198237666637962215",Service="Time Exceeded (TTL Exceeded)"^J
$ cat line.log | lognormalizer -rruleBase.rb 
{ "rest": "Facility=\"Packet Filtering\",Type=\"Notification\",Event=\"Connection discarded\",Action=\"Discard\",Protocol=\"1\",Src=\"10.0.0.10\",Dst=\"10.10.10.10\",RuleId=\"21.0\",Srcif=\"0\",IcmpType=\"11\",IcmpCode=\"0\",CompId=\"Limes node 1\",ReceptionTime=\"2024-05-20 10:25:42\",SenderType=\"Firewall\",SituationId=\"70019\",Situation=\"Connection_Discarded\",EventId=\"7198237666637962215\",Service=\"Time Exceeded (TTL Exceeded)\"^J", "customfromhost": "192.168.1.100", "logid": "535666280", "timestamp": "2024-05-20 10:25:42", "logsender": "192.168.20.100", "logtime": "May 20 10:25:42" }

The installed rsyslog is the default version for CentOS-STREAM9:

# dnf info rsyslog
Last metadata expiration check: 0:11:45 ago on Mon 20 May 2024 12:57:06 PM CEST.
Installed Packages
Name         : rsyslog
Version      : 8.2310.0
Release      : 4.el9
Architecture : x86_64
Size         : 2.6 M
Source       : rsyslog-8.2310.0-4.el9.src.rpm
Repository   : @System
From repo    : appstream

What am I doing wrong?

Trey · Accepted Answer · 2024-06-06 09:47:22Z

I have searched online for a long time but haven't found many examples related to mmnormalize.

this is my syslog

2024-06-04 12:09:14.209354 1717474154 QID="35007" identity="disabled" family="" protocol="1" rbytes="38" rtime="1717474154.159939" srccountry="00" srcip="192.168.10.13" srcport="25942" dstip="192.168.10.111" dstport="53" ndomain="0" qtype="TXT" qname="cisco.com." rcode="NOERROR" rrcount="14" category="Information Technology" ctype="2" ctrigger="cisco.com." chit="cisco.com" caction="1" tags="" qtime="1717474154.159920" duration="0" valid="1" rname="cisco.com." rtype="TXT" rttl="900" nresponse="0" dstcountry="" rdata=""test""

My rsyslog configuration setup

# /etc/rsyslog.d/20-identity.conf

# Load necessary modules
module(load="imfile")  # Allow reading logs from a file
module(load="mmnormalize")  # Load mmnormalize module

# Define a file input
input(type="imfile"
      File="/var/log/query.log"
      Tag="querylog"
      Severity="info"
      Facility="local7")

# Use normalize rules to parse logs
action(type="mmnormalize" 
ruleBase="/etc/rsyslog.d/normalize.rulebase")

# Define a template to parse syslog messages and add custom strings and newlines at the end
template(name="IdentityTemplate1" type="string" string="%msg%\n")
template(name="IdentityTemplate2" type="string" string="%msg% keyword=\"Family\"\n")

# Define a rule to check if `identity` is `disable`
if ($!identity == "disabled" and $syslogtag == 'querylog') then {
    # Write matching messages to a specific log file
    action(type="omfile" file="/var/log/A-identity-enabled.log" template="IdentityTemplate1")
}

# Define a rule to check if `protocol` is `1`
if ($!protocol == "1" and $syslogtag == 'querylog') then {
    # Write matching messages to a specific log file
    action(type="omfile" file="/var/log/B-protocol-enabled.log" template="IdentityTemplate2")
}

# Capture all remaining log messages to syslog
#*.* /var/log/syslog

#Skip recording all remaining log messages to syslog
stop

The file of rulebase

#/etc/rsyslog.d/normalize.rulebase
version=2

rule=:%timestamp:char-to: .% %time:char-to: % %epoch:number% QID=%QID:op-quoted-string% identity=%identity:op-quoted-string% family=%family:op-quoted-string% protocol=%family:op-quoted-string% %rest:rest%

You can use the command:

$ rsyslogd -N1

to verify whether the rsyslog configuration file syntax is correct

Touch the empty log file

$ touch /var/log/query.log

When the setup is completed, you need to restart the rsyslog service.

$ sudo systemctl restart rsyslog.service

Then you can test whether the rule functions properly by using the following command:

$ echo "2024-06-03 08:58:49.529820 1717376329 QID=\"39152\" identity=\"disabled\" family=\"1\" protocol=\"1\" rbytes=\"38\" rtime=\"1717376329.168113\" srccountry=\"00\" srcip=\"192.168.10.13\"" >> /var/log/query.log

This will check if the logs are correctly written to /var/log/A-identity-enabled.log and /var/log/B-protocol-enabled.log.

#/var/log/A-identity-enabled.log
2024-06-03 08:58:49.529820 1717376329 QID="39152" identity="disabled" family="1" protocol="1" rbytes="38" rtime="1717376329.168113" srccountry="00" srcip="192.168.10.13"


#/var/log/B-protocol-enabled.log
2024-06-03 08:58:49.529820 1717376329 QID="39152" identity="disabled" family="1" protocol="1" rbytes="38" rtime="1717376329.168113" srccountry="00" srcip="192.168.10.13" keyword="Family"

Reference: https://github.com/rsyslog/liblognorm/blob/v2-experimental/tests/field_whitespace_v1.sh

https://rsyslog.github.io/liblognorm/doc/_build/html/sample_rulebase.html

Stack Exchange Network

rsyslog variable from mmnormalize as part of omfile filename

1 Answer 1

You must log in to answer this question.

Hot Network Questions

rsyslog variable from mmnormalize as part of omfile filename

1 Answer 1

You must log in to answer this question.

Related

Hot Network Questions