Sync data between Mongodb and ElasticSearch using Logstash

Question

I am trying to sync my MongoDB database with elasticsearch and use the data on Kibana. I am using the plugins logstash-input-mongoDb and logstash-output-elasticsearch.

The issue is in the configuration file in Logstash to connect to MongoDB. Once I setup both plugins I created a config file to mention the input source details (MongoDB) and output (ElasticSearch). I have written it here:

input {
       uri => 'mongodb://localhost:27017/t_data/data'
       placeholder_db_dir => '/opt/logstash-mongodb/'
       placeholder_db_name => 'logstash_sqlite.db'
       collection => 't_data'
       batch_size => 5000
}

filter {
}

output {
        stdout {
                codec => rubydebug
       }
       elasticsearch {
               action => "index"
               index => "mongo_t_data"
               hosts => ["localhost:9200"]
       }
}

I then run this command from the bin directory in logstash where the config file is saved as mongodata.conf:

 \logstash-7.10.1\bin>logstash -f mongodata.conf

I obtain the following error:

 [2021-01-17T00:11:22,730][ERROR][logstash.agent           ] Failed to execute action 
 {:action=>LogStash::PipelineAction::Create/pipeline_id:main, 
 :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"{\" at 
 line 2, column 13 (byte 22) after input {\r\n        uri ", :backtrace=>["C:/PESU/logstash-7.10.1- 
 windows-x86_64/logstash-7.10.1/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", 
 "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", 
 "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "C:/PESU/logstash-7.10.1- 
 windows-x86_64/logstash-7.10.1/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", 
 "C:/PESU/logstash-7.10.1-windows-x86_64/logstash-7.10.1/logstash- 
 core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "C:/PESU/logstash-7.10.1-windows- 
 x86_64/logstash-7.10.1/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]}
 [2021-01-17T00:11:23,310][INFO ][logstash.agent           ] Successfully started Logstash API 
 endpoint {:port=>9600}
 [2021-01-17T00:11:28,026][INFO ][logstash.runner          ] Logstash shut down.
 [2021-01-17T00:11:28,058][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: 
 Logstash stopped processing because of an error: (SystemExit) exit

I have attached the screenshot here:

My goal is to get logstash connected to Mongodata and elasticsearch succesfully to be able to sync data and add documents when required. I suspect it is an issue with me using Notepad to create the .conf file.

R2D2 · Accepted Answer · 2021-01-17 09:26:48Z

2

MongoDB since version 3.6 has feature called changesteams that can be used for this task, check here : https://medium.com/@montumodi/realtime-sync-from-mongodb-to-elasticsearch-using-change-streams-2559ad312011

answered Jan 17, 2021 at 9:26

R2D2

10.9k2 gold badges20 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

leandrojmp · Accepted Answer · 2021-01-16 19:35:38Z

0

Your input config is wrong. It is missing the type of input plugin.

It should be something like this:

input {
    mongodb {
        uri => 'mongodb://localhost:27017/t_data/data'
        placeholder_db_dir => '/opt/logstash-mongodb/'
        placeholder_db_name => 'logstash_sqlite.db'
        collection => 't_data'
        batch_size => 5000
    }
}

Try to change and see if it works.

edited Jan 16, 2021 at 19:35

answered Jan 16, 2021 at 19:14

leandrojmp

7,5122 gold badges21 silver badges24 bronze badges

1 Comment

B2A3R9C9A Over a year ago

looks that that was part of the issue, thanks. Would you recommend any other plugins useful to sync data between Elasticsearch(Kibana) and MongoDB. I have looked into different Plugins and Extensions but went with Logstash as I read that one of its purpose is to act as connector in the ELK stack. I am very new to this so would like more info

Collectives™ on Stack Overflow

Sync data between Mongodb and ElasticSearch using Logstash

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related