How can i enable dynamic scripting in AWS ES?

Question

I am using aws elastic service and indexed 650 000 data. I need to add two new fields to the already indexed documents. When I tried to call the updateByQuery function got the error, 'scripts of type [inline], operation [update] and lang [groovy] are disabled'. I have fixed it by adding
script.engine.groovy.inline.aggs: on script.engine.groovy.inline.update: on on elasticsearch.yml and it works perfectly on local . How can I add this configuration on aws es ? I am getting the same error when I am updating documents in aws elastic service.

Here is my code. I want to update all records ( where "device"= deviceVal) by adding new fields Site and Time.

var site = 'some value';
var deviceVal = '123'; 

var theScript = {
    "inline": "ctx._source.Site = '"+ site + "';ctx._source.Time = '"+ new Date().getTime() + "'"
}   
var match = {
    "match": { "device": deviceVal }
}

client.updateByQuery({
    index: 'my_index',
    type:'txt',

    "body": {
        "query": match, 
        "script":theScript
    }

}, function (error, response) {
    // console.log("success")
    console.log('error--',error)
    console.log('response--',response)
});

This answer should help: stackoverflow.com/questions/38987919/… (hint: the _update_by_query endpoint is not supported by the AWS ES Service). You're better off reindexing your data into a new index. — Val
– Val, Commented Nov 23, 2016 at 15:03
@Val , How can I add new fields with dynamic value while reindexing with logstash ? . I need to add a new field 'Site' to the document and The value of 'Site' depends on the 'device' field which is already in document. — Jennifer
– Jennifer, Commented Nov 24, 2016 at 4:20
@Val , How can I add new fields with dynamic value while reindexing with logstash ? . I need to add two new fields 'Site' & 'Location' to the document. The values of 'Site' & 'Location' depends on the 'device' field which is already in document. — Jennifer
– Jennifer, Commented Nov 24, 2016 at 4:27

Community · Accepted Answer · 2017-05-23 11:47:23Z

2

Building on the other answer where we use logstash to reindex into an AWS ES cluster, you simply need to add one more transformation where # add other transformations here is mentioned.

In your case the input part needs to contain a query for the device:

input {
  elasticsearch {
   hosts => ["my-elasticsearch-domain.us-west-2.es.amazonaws.com:80"]
   index => "my_index"
   query => '{"query": {"match":{"device": "123"}}}'
   docinfo => true
  }
}

And the filter part would boil down to this, i.e. we rename the @timestamp field and add the Site field:

filter {
 mutate {
  remove_field => [ "@version" ]
  rename => { "@timestamp" => "Time" }
  add_field => { "Site" => "some value" }
 }
}

edited May 23, 2017 at 11:47

CommunityBot

11 silver badge

answered Nov 24, 2016 at 4:31

Val

218k14 gold badges377 silver badges384 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Jennifer Over a year ago

Thanks for your help. If I have multiple values for device like 123, 456,789 etc then can I reindex all documents in single step ? The value of Site changed with device. Or do I need to execute the logstash script multiple times for each device value ?

Val Over a year ago

If you only have a dozen or so devices to update, I'd suggest running this manually a couple times. If we're talking a few dozens, hundreds or more, then the above solution can be automatized, the simplest being with a shell script that generates the logstash config and runs logstash for every configuration.

Jennifer Over a year ago

Thank you very much Val. I think shell script is a nice idea.

Jennifer Over a year ago

How can I set environment variables in logstash? I have tried with shell script. Set the value as follows. export site="Hotel California" And in my .conf file filter { mutate { remove_field => [ "@version"] add_field => { "Site" => "${site}" } } } But the field site is not updated from the environent variable. Its value saved as "${site}".

Collectives™ on Stack Overflow

How can i enable dynamic scripting in AWS ES?

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related