Index not creating in kibana for mongodb + logstash +elasticsearch

Question

I'm trying to visualize the mongodb data in kibana using logstash configuration.Below is my configuration.I'm getting some outputs in terminal and it is looping forever. I couldn't see any index created by the name mentioned in the config file and if the index was generated also don't have any data on it. Saying no results to match in the discover tab.How to make the configuration to visualize the data in kibana?

 input {
            mongodb {
            uri => "mongodb+srv:@@@@@@@@@@@@@@@?retryWrites=true&w=majority"
            placeholder_db_dir => "C:/logstash-mongodb"
            placeholder_db_name => "logstash1_sqlite.db"
            collection => "logs"
            batch_size => 1
            
            
            }
    }
    filter {
    
    }
    output {
            stdout {
                    codec => rubydebug
            }
            elasticsearch {
                    action => "index"
                    index => "ayesha_logs"
                    hosts => ["localhost:9200"]
            }
    }

http://localhost:9200/ayesha_logs/_search?pretty

Terminal logs: D, [2020-10-01T08:11:45.717000 #2372] DEBUG -- : MONGODB | range-api-test-cluster-shard-00-02.icqif.azure.mongodb.net:27017 req:259 conn:1:1 sconn:231839 | coexistence-poc.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "$db"=>"coexistence-poc", "$clusterTime"=>{"clusterTime"=>#<BSON::Timestamp:0x32598cb2 @increment=1, @seconds=1601532700>, "signature"=>{"hash"=><BSON::Binary:0x2622 type=generic data=0xfaf25a8d85... D, [2020-10-01T08:11:45.755000 #2372] DEBUG -- : MONGODB | range-api-test-cluster-shard-00-02.icqif.azure.mongodb.net:27017 req:259 | coexistence-poc.listCollections | SUCCEEDED | 0.038s D, [2020-10-01T08:11:50.801000 #2372] DEBUG -- : MONGODB | range-api-test-cluster-shard-00-02.icqif.azure.mongodb.net:27017 req:260 conn:1:1 sconn:231839 | coexistence-poc.find | STARTED | {"find"=>"coexistence-pinfobackfill-logs", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5f71f009b6b9115861d379d8')}}, "limit"=>50, "$db"=>"coexistence-poc", "$clusterTime"=>{"clusterTime"=>#<BSON::Timestamp:0x32598cb2 @increment=1, @seconds=1601532700>, ... D, [2020-10-01T08:11:50.843000 #2372] DEBUG -- : MONGODB | range-api-test-cluster-shard-00-02.icqif.azure.mongodb.net:27017 req:260 | coexistence-poc.find | SUCCEEDED | 0.042s D, [2020-10-01T08:11:50.859000 #2372] DEBUG -- : MONGODB | range-api-test-cluster-shard-00-02.icqif.azure.mongodb.net:27017 req:261 conn:1:1 sconn:231839 | coexistence-poc.listCollections | STARTED | {"listCollections"=>1, "cursor"=>{}, "nameOnly"=>true, "$db"=>"coexistence-poc", "$clusterTime"=>{"clusterTime"=>#<BSON::Timestamp:0x32598cb2 @increment=1, @seconds=1601532700>, "signature"=>{"hash"=><BSON::Binary:0x2622 type=generic data=0xfaf25a8d85... D, [2020-10-01T08:11:50.906000 #2372] DEBUG -- : MONGODB | range-api-test-cluster-shard-00-02.icqif.azure.mongodb.net:27017 req:261 | coexistence-poc.listCollections | SUCCEEDED | 0.047s

Jaycreation · Accepted Answer · 2020-10-01 05:45:08Z

1

Did you create your Kibana's index pattern ? If not, just go to Menu > stack managment > Kibana > Index pattern

click on

And follow the steps.

You will then be able to use you index in Discover or visualization tabs.

answered Oct 1, 2020 at 5:45

Jaycreation

2,0891 gold badge15 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Ayesha Over a year ago

I'm able to create the index but while visualizing i couldn't see any data on it. Please see the screenshot which have added now.

Jaycreation Over a year ago

1 - in the dev tools , query you index with GET myindex/_count to verify that it's well populated. 2- do the same with a _search ordered by @timestamp desc an check the last date of insert. 3- verify your time range in kibana

Ayesha Over a year ago

GET ayesha_logs/_count :{ "count" : 0, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 } }

Jaycreation Over a year ago

You don't have no data in your index. so the problem is not on the kibana side. The issue is at the logstash side. I did not seen the problem in the logs you post, are you sure the your mongodb query is finding documents ?

Ayesha Over a year ago

I could see the one of mongodb document id in terminal.coexistence-poc.find | STARTED | {"find"=>"coexistence-pinfobackfill-logs", "filter"=>{"_id"=>{"$gt"=>BSON::ObjectId('5f71f009b6b9115861d379d8')}}, "limit"=>50, "$db"=>"coexistence-poc", "$clusterTime"=>{"clusterTime"=>#<BSON::Timestamp:0x32598cb2 @increment=1, @seconds=1601532700>,

|

Collectives™ on Stack Overflow

Index not creating in kibana for mongodb + logstash +elasticsearch

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related