Elasticsearch have duplicate record

Question

I'm using ELK stack on Bitnami service and using Serilog to send log to Logstash but when I send one log, In kibana I will see two record with different _id a the same message.

This is my logstash file. One more question, my log model have Id field, this is unique. How can I set it to document_id.

You need to share your input and filter configs to help understand what is happening in your pipeline. — leandrojmp
– leandrojmp, Commented Jul 17, 2019 at 14:25
And your inputs? How are you receiving the logs? How is the data? You need to provide more information about what are you doing and what you'd expect the result to be. — leandrojmp
– leandrojmp, Commented Jul 18, 2019 at 0:52
I'm using serilog and config it to send log to logstash. The log model is just a simple model with Id field is a Guid. — Bui Quang Huy
– Bui Quang Huy, Commented Jul 18, 2019 at 1:53
I'm following this tutorial: blog.romanpavlov.me/logging-serilog-elk — Bui Quang Huy
– Bui Quang Huy, Commented Jul 18, 2019 at 1:54

Badger · Accepted Answer · 2019-07-17 14:43:24Z

1

If you have a unique id in a field you can use the document_id option on the elasticsearch output with a sprintf reference to the field

document_id => "%{Id}"

answered Jul 17, 2019 at 14:43

Badger

4,1272 gold badges9 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Bui Quang Huy Over a year ago

I tried but may be %{Id} not get my Id Field to the document_id

Collectives™ on Stack Overflow

Elasticsearch have duplicate record

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related