Elasticsearch | Mapping exclude field with bulk API

Question

I am using bulk api to create index and store data fields. Also I want to set mapping to exclude a field "field1" from the source. I know this can be done using "create Index API" reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html but I am using bulk API. below is sample API call:

POST _bulk

{ "index" : { "_index" : "test", _type = 'testType', "_id" : "1" } }
{ "field1" : "value1" }

Is there a way to add mapping settings while bulk indexing similar to below code:

  { "index" : { "_index" : "test", _type = 'testType', "_id" : "1" },
     "mappings": {
          "_source": {
               "excludes": [
                             "field1"
                           ]
          }
     }
  }
  { "field1" : "value1" }

how to do mapping with bulk API?

samy · Accepted Answer · 2021-10-26 10:59:48Z

1

It is not possible to define the mapping for a new Index while using the bulk API. You have to create your index beforehand and define the mapping then, or you have to define an index template and use a name for your index in your bulk request that triggers that template.

The following example code can be executed via the Dev Tools windows in Kibana:

PUT /_index_template/mytemplate
{
  "index_patterns": [
    "te*"
  ],
  "priority": 1,
  "template": {
    "mappings": {
      "_source": {
        "excludes": [
          "testexclude"
        ]
      },
      "properties": {
        "testfield": {
          "type": "keyword"
        }
      }
    }
  }
}

POST _bulk
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "testfield" : "value1", "defaultField" : "asdf", "testexclude": "this shouldn't be in source" }

GET /test/_mapping

You can see by the response that in this example the mapping template was used for the new test index because the testfield has only the keyword type and the source excludes is used from the template.

{
  "test" : {
    "mappings" : {
      "_source" : {
        "excludes" : [
          "testexclude"
        ]
      },
      "properties" : {
        "defaultField" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "testexclude" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "testfield" : {
          "type" : "keyword"
        }
      }
    }
  }
}

Also the document is not returned with the excluded field:

GET /test/_doc/1

Response:

{
  "_index" : "test",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : 1,
  "_seq_no" : 0,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "defaultField" : "asdf",
    "testfield" : "value1"
  }
}

Hope this answers your question and solves your use-case.

edited Oct 26, 2021 at 10:59

answered Oct 25, 2021 at 10:46

samy

1,4402 gold badges20 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Manoj Saini Over a year ago

I tried your solution but got an error. Solution I applied using Elasticsearch 7.7. Any idea what is this error? PUT _template/template_manoj { "index_patterns": ["manoj-*"], "mappings": { "_source": { "excludes": [ "@message" ] } } } { "index" : { "_index" : "manoj-2021-10-26", "_id" : "1" } } { "testfield" : "value1", "@message": "this shouldn't be in source" }

Manoj Saini Over a year ago

Error: { "errorType": "Error", "errorMessage": "{\"statusCode\":200,\"responseBody\":{\"took\":312,\"errors\":true}}", "stack": [ "Error: {\"statusCode\":200,\"responseBody\":{\"took\":312,\"errors\":true}}", " at _homogeneousError (/var/runtime/CallbackContext.js:12:12)", " at postError (/var/runtime/CallbackContext.js:29:54)", " at done (/var/runtime/CallbackContext.js:58:7)", " at fail (/var/runtime/CallbackContext.js:70:7)",

Manoj Saini Over a year ago

" at Object.fail (/var/runtime/CallbackContext.js:106:16)", " at /var/task/index.js:43:25", " at IncomingMessage.<anonymous> (/var/task/index.js:187:13)", " at IncomingMessage.emit (events.js:326:22)", " at endReadableNT (_stream_readable.js:1241:12)", " at processTicksAndRejections (internal/process/task_queues.js:84:21)" ] }

samy Over a year ago

Hi i ran these examples via the dev tools window in Kibana. In what environment (elasticsearch client) are you trying to create that template? Looks to me like node environment, i have no experience with that, you have to check how you create a put requests that sends that template to Elasticsearch in that client environment. I added the information that the example code is kibana Dev Tools specific to the answer.

Manoj Saini Over a year ago

Hi I am using Kibana dev tool for template PUT _template/template_manoj { "index_patterns": ["manoj-*"], "mappings": { "_source": { "excludes": [ "@message" ] } } }

|

Collectives™ on Stack Overflow

Elasticsearch | Mapping exclude field with bulk API

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related