Why Elasticsearch replaces document instead of create new?

Question

ES newbie here. We have an index with non-dynamic mapping. When I put a document, it replaces existing one (I see big number in "_version") instead of adding new. I don't explicitly define any doc id, should I generate a random one?

Document sample:

{
    "environment": "acl-distributiondev1_logs-develop",
    "log_event": {
        "full_message": "START RequestId: da8fa922-8cb4-4e24-9aa0-654f06b28cfe Version: $LATEST\n",
        "id": "34933837981111111111111158539671171070636938547602391040",
        "request_id": "da8fa922-8cb4-4e24-9aa0-654f06b28cfe",
        "timestamp": 1566487472539,
        "type": "START"
    },
    "log_group": "/aws/lambda/function",
    "log_stream": "2019/08/22/[$LATEST]0b98ccbf0a1111b989ff1707b285a78b",
    "log_stream_id": "0b98ccbf0a1111b989ff1707b285a78b",
    "owner": "024031111114",
    "record_timestamp": 1566487473520,
    "region": "eu-west-1"
}

Mapping:

{
  "mappings": {
    "_doc": {
      "properties": {
        "log_stream_id": {
          "type": "keyword"
        },
        "region": {
          "type": "keyword"
        },
        "record_timestamp": {
          "type": "date"
        },
        "owner": {
          "type": "text"
        },
        "log_group": {
          "type": "keyword"
        },
        "log_stream": {
          "type": "text"
        },
        "environment": {
          "type": "keyword"
        },

        "log_event": {
          "properties": {
            "id": {
              "type": "keyword"
            },
            "request_id": {
              "type": "keyword"
            },
            "timestamp": {
              "type": "date"
            },
            "type": {
              "type": "keyword"
            },
            "full_message": {
              "type": "text"
            }
          }
        }
      }
    }
  }
}

What am I doing wrong?

Rob · Accepted Answer · 2019-08-22 18:26:55Z

If you don't specify id in indexing url elasticsearch will autogenerate id for a document for you. Documents won't be overwritten.

Index document request:

POST tmp/_doc
{
    "environment": "acl-distributiondev1_logs-develop",
    "log_event": {
        "full_message": "START RequestId: da8fa922-8cb4-4e24-9aa0-654f06b28cfe Version: $LATEST\n",
        "id": "34933837981111111111111158539671171070636938547602391040",
        "request_id": "da8fa922-8cb4-4e24-9aa0-654f06b28cfe",
        "timestamp": 1566487472539,
        "type": "START"
    },
    "log_group": "/aws/lambda/function",
    "log_stream": "2019/08/22/[$LATEST]0b98ccbf0a1111b989ff1707b285a78b",
    "log_stream_id": "0b98ccbf0a1111b989ff1707b285a78b",
    "owner": "024031111114",
    "record_timestamp": 1566487473520,
    "region": "eu-west-1"
}

Response:

{
  "_index" : "tmp",
  "_type" : "_doc",
  "_id" : "sB-PumwBD4taKfEdJjeK", <-- generated by elasticsearch
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 4,
  "_primary_term" : 1
}

Doing the same request returns

{
  "_index" : "tmp",
  "_type" : "_doc",
  "_id" : "sR-QumwBD4taKfEdrDeG", <-- new id generated by elasticsearch
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 5,
  "_primary_term" : 1
}

GET tmp/_count returns

{
  "count" : 2,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  }
}

Maybe you are just passing id in URL by accident and all of the requests are replacing the single document.

Hope that helps.

Collectives™ on Stack Overflow

Why Elasticsearch replaces document instead of create new?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related