logstash configuration with variable index name for elasticsearch

Question

I have a logstash config which gets the data from redis and outputs the data to elasticsearch. This is what my configuration file looks like:

input {
    redis {
        host => "127.0.0.1"
        codec => "json"
        key => "logstash"
        data_type => "list"
    }
}

output {
    elasticsearch {
        protocol => "http"
        user => "user"
        password => "password"
        host => "host:9200"
        index => "index-%{foo}"
        document_id => "id-%{bar}"
        document_type => "my_type"
        cluster => "my_cluster"
    }
}

My elasticsearch instance is proxied on port 9200 by nginx, in reality it runs on a different port. Also the auth is set using nginx, not shield. The problem is that when I run a configtest on this config, it shows Configuration OK. But when I run this script then this error is thrown:

Pipeline aborted due to error {:exception=>"LogStash::ConfigurationError", 
:backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-
2.4.0-java/lib/logstash/config/mixin.rb:88:in `config_init'", 
"org/jruby/RubyHash.java:1342:in `each'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-
java/lib/logstash/config/mixin.rb:72:in `config_init'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-
java/lib/logstash/outputs/base.rb:79:in `initialize'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-
java/lib/logstash/output_delegator.rb:74:in `register'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-
java/lib/logstash/pipeline.rb:181:in `start_workers'", 
"org/jruby/RubyArray.java:1613:in `each'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-
java/lib/logstash/pipeline.rb:181:in `start_workers'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-
java/lib/logstash/pipeline.rb:136:in `run'", 
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-
java/lib/logstash/agent.rb:491:in `start_pipeline'"], :level=>:error}

Why is this error thrown? What am I doing wrong here?

EDIT

Note that I've also tried using

host => "http://user:password@host:9200"

but this doesn't work either.

baudsp · Accepted Answer · 2016-09-07 08:47:14Z

1

If you are using Logstash version 2+, it's not host but hosts.

Cf documentation

hosts
Value type is array
Default value is ["127.0.0.1"]
Sets the host(s) of the remote instance.

edited Sep 7, 2016 at 8:47

answered Sep 7, 2016 at 8:34

baudsp

4,1091 gold badge20 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Keshav Agarwal Over a year ago

That works. One question though, if it is mentioned in the documentation, then why did configtest showed Configuration OK?

baudsp Over a year ago

@KeshavAgarwal it's a guess, but I think configtest only checks if the configuration syntax is correct, but not the configuration options of the filters/outputs/inputs

Collectives™ on Stack Overflow

logstash configuration with variable index name for elasticsearch

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related