1

I am trying to capture syslog messages sent over the network using rsyslog, and then have rsyslog capture, transform and send these messages to elasticsearch.

I found a nice article on the configuration on https://www.reddit.com/r/devops/comments/9g1nts/rsyslog_elasticsearch_logging/

Problem is that rsyslog keeps popping up an error at startup that it cannot connect to Elasticsearch on the same machine on port 9200. Error I get is Failed to connect to localhost port 9200: Connection refused

2020-03-20T12:57:51.610444+00:00 53fd9e2560d9 rsyslogd: [origin software="rsyslogd" swVersion="8.36.0" x-pid="1" x-info="http://www.rsyslog.com"] start

rsyslogd: omelasticsearch: we are suspending ourselfs due to server failure 7: Failed to connect to localhost port 9200: Connection refused [v8.36.0 try http://www.rsyslog.com/e/2007 ]

Anyone can help on this?

Everything is running in docker on a single machine. I use below docker compose file to start the stack.

version: "3"

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.6.1
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
    ports:
      - 9200:9200
    networks:
      - logging-network

  kibana:
    image: docker.elastic.co/kibana/kibana:7.6.1
    depends_on:
      - logstash
    ports:
      - 5601:5601
    networks:
      - logging-network

  rsyslog:
    image: rsyslog/syslog_appliance_alpine:8.36.0-3.7
    environment:
      - TZ=UTC
      - xpack.security.enabled=false
    ports:
      - 514:514/tcp
      - 514:514/udp
    volumes:
      - ./rsyslog.conf:/etc/rsyslog.conf:ro
      - rsyslog-work:/work
      - rsyslog-logs:/logs

volumes:
  rsyslog-work:
  rsyslog-logs:

networks:
  logging-network:
    driver: bridge

rsyslog.conf file below:

global(processInternalMessages="on")

#module(load="imtcp" StreamDriver.AuthMode="anon" StreamDriver.Mode="1")
module(load="impstats") # config.enabled=`echo $ENABLE_STATISTICS`)
module(load="imrelp")
module(load="imptcp")
module(load="imudp" TimeRequery="500")

module(load="omstdout")
module(load="omelasticsearch")

module(load="mmjsonparse")
module(load="mmutf8fix")


input(type="imptcp" port="514")
input(type="imudp" port="514")
input(type="imrelp" port="1601")

# includes done explicitely
include(file="/etc/rsyslog.conf.d/log_to_logsene.conf" config.enabled=`echo $ENABLE_LOGSENE`)
include(file="/etc/rsyslog.conf.d/log_to_files.conf" config.enabled=`echo $ENABLE_LOGFILES`)



#try to parse a structured log
action(type="mmjsonparse")

# this is for index names to be like: rsyslog-YYYY.MM.DD
template(name="rsyslog-index" type="string" string="rsyslog-%$YEAR%.%$MONTH%.%$DAY%")

# this is for formatting our syslog in JSON with @timestamp
template(name="json-syslog" type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"host\":\"")        property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"program\":\"")     property(name="programname")
      constant(value="\",\"tag\":\"")         property(name="syslogtag" format="json")
      constant(value="\",")                   property(name="$!all-json" position.from="2")
    # closing brace is in all-json
}

# this is where we actually send the logs to Elasticsearch (localhost:9200 by default)
action(type="omelasticsearch" template="json-syslog" searchIndex="rsyslog-index" dynSearchIndex="on")



#################### default ruleset begins ####################

# we emit our own messages to docker console:
syslog.* :omstdout:

include(file="/config/droprules.conf" mode="optional")  # this permits the user to easily drop unwanted messages

action(name="main_utf8fix" type="mmutf8fix" replacementChar="?")

include(text=`echo $CNF_CALL_LOG_TO_LOGFILES`)
include(text=`echo $CNF_CALL_LOG_TO_LOGSENE`)

1 Answer 1

1

First of all you need to run all the containers on the same docker network which in this case are not. Second , after running the containers on the same network , login to rsyslog container and check if 9200 is available.

Sign up to request clarification or add additional context in comments.

3 Comments

+1 , I was also thinking on the same lines, but i am not sure how to run all dockers in same network :(, can you refer good docs or commands to do that?
I have sent you a proposal on upwork , i can fix it , but i need access to your environment and need some live debugging
Thanks for this quick suggestion. I did add the rsyslog on the same network, and tried if the port is open from the rsyslog container using "nc -zc elasticsearch 9200" and it reports as 'open'. Unfortunately the rsyslog still does not want to connect..

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.