Unable to index with docker logstash

Question

I am using the latest code of [email protected]:deviantony/docker-elk.git repository to host ELK stack with docker-compose up command. Elastic search and kibana are running fine.

Although I cannot index into logstash with my logstash.conf which is as shown below:

input {
    file {
        # Configure your path below
        path => ["C:/Users/matt/Desktop/temp/logs/*.txt*"]
        ignore_older => "141 days"
        start_position => "beginning"
        file_sort_by => "last_modified"
        file_sort_direction => "desc"
        sincedb_path => "NUL"
        type => "appl"
        codec => multiline {
            pattern => "^<log4j:event"
            negate => true
            what => "previous"
        }
    }
}
filter {
    if [type] == "appl" {
        grok {
            add_tag => [ "groked" ]
            match => ["message", ".*"]
            remove_tag => ["_grokparsefailure"]
        }
        mutate {
            gsub => ["message", "log4j:", ""]
        }
        xml {
            source => "message"
            remove_namespaces => true
            target => "log4jevent"
            xpath => [ "//event/@timestamp", "timestamp" ]
            xpath => [ "//event/@level", "loglevel" ]
            xpath => [ "/event/message/text()", "message" ]
            xpath => [ "/event/throwable/text()", "exception" ]
            xpath => [ "//event/properties/data[@name='log4jmachinename']/@value", "machinename" ]
            xpath => [ "//event/properties/data[@name='log4japp']/@value", "app" ]
            xpath => [ "//event/properties/data[@name='log4net:UserName']/@value", "username" ]
            xpath => [ "//event/properties/data[@name='log4net:Identity']/@value", "identity" ]
            xpath => [ "//event/properties/data[@name='log4net:HostName']/@value", "hostname" ]
        }
        mutate {
            remove_field => ["type"]
            gsub => [
            "message", "&amp;", "&",
            "message", "&lt;", "<",
            "message", "&gt;", ">",
            "message", "&quot;", "\"",
            "message", "&apos;", "'"
            ]
        }
        date {
            match => [ "[timestamp][0]","UNIX_MS" ]
            target => "@timestamp"
            remove_field => ["timestamp"]
        }
    }
}
output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "log4jevents"
        user => "elastic"
        password => "changeme"
        ecs_compatibility => disabled
    }
    stdout {
        codec => rubydebug
    }
}

and my log file that I want to index with my logstash is shown below

<log4j:event logger="Microsoft.Unity.ApplicationBlocks.Logging.Logger" timestamp="1615025506621" level="DEBUG" thread="13"><log4j:message>SSO-&gt;AccountController-&gt;Login-&gt;Before ClientID Check</log4j:message><log4j:properties><log4j:data name="log4jmachinename" value="hostname01" /><log4j:data name="log4japp" value="/LM/W3SVC/2/ROOT-1-132594985694777790" /><log4j:data name="log4net:UserName" value="IIS APPPOOL\default" /><log4j:data name="log4net:Identity" value="" /><log4j:data name="log4net:HostName" value="hostname01" /></log4j:properties><log4j:locationInfo class="Microsoft.Unity.ApplicationBlocks.Logging.Logger" method="Debug" file="F:\somefolder\Agent\_work\1\s\Unity\Microsoft.Unity.ApplicationBlocks\Logging\Logging.cs" line="353" /></log4j:event>

The issue shown while starting the docker-compose up is shown below for logstash

Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elastic:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elastic:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

The same logstash.conf was working earlier for EK version 6.8. Whats wrong with my logstash.conf?

glenacota · Accepted Answer · 2021-03-20 16:47:41Z

2

In your output elasticsearch plugin, set the hosts property to elasticsearch:9200.

output {
    elasticsearch {
        hosts => ["elasticsearch:9200"]
        index => "log4jevents"
        user => "elastic"
        password => "changeme"
        ecs_compatibility => disabled
    }
    stdout {
        codec => rubydebug
    }
}

answered Mar 20, 2021 at 16:47

glenacota

2,5571 gold badge14 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Garrison Patton Over a year ago

thank you the above issue is resolved. However I am still getting some error. ` Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"{\" at line 1, column 6 (byte 6) after input", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `

Collectives™ on Stack Overflow

Unable to index with docker logstash

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related