Elastic Search Bulk Update using python, how to append a array field with new data

Question

how to update field in elastic search using bulkupdate in python. i tried many ways its all getting error. In some cases i am getting document missing error , how do i update and upsert at same time . and also appending to field is not working.elasticsearch==7.9.1 is the package i used in python

for i in range(0, length, steps):
    end_index = length-1 if i+steps>length else i+steps
    temp_list = test_data[i: end_index]
    bulk_file = ''
    actions = [{
        "_index": "test-data",
        "_opt_type":"update",
        "_type": "test-test-data",
        "_id": test_row ['testId'],
        "doc":{"script": {
                          "source": "ctx._source.DataIds.add(params.DataIds)",
                          "lang": "painless",
                          "params": {
                              "DataIds":test_row ['DataIds']
                          }
                      }}
        }
        for test_row in temp_list
    ]
    helpers.bulk(es, actions)

Error iam getting is this

    {'update': {'_index': 'test-data', '_type': 'products', '_id': '333', 'status': 400, 'error': {'type': 'illegal_argument_exception', 'reason': 'failed
 to execute script', 'caused_by': {'type': 'script_exception', 'reason': 'runtime error', 'script_stack': ['ctx._source.dataIds.add(params.dataIds)', '
    ^---- HERE'], 'script': 'if (ctx._source.dataIds == null) { ctx._source.dataIds = []; } ctx._source.dataIds.add(params.dataIds)', 'lang': 'painless', 'position': {'offse
t': 105, 'start': 71, 'end': 118}, 'caused_by': {'type': 'illegal_argument_exception', 'reason': 'dynamic method [java.lang.String, add/1] not found'}}}, 'data': {'upsert': {}, 'scripted_up
sert': True, 'script': {'source': 'if (ctx._source.dataIds == null) { ctx._source.dataIds = []; } ctx._source.dataIds.add(params.dataIds)', 'lang': 'painless', 'params': {'c
dataIds': 'set123'}}}}}])

@Lupanoide when i change to doc field, fields in the elastic search are added like this "doc.script.lang painless". iam not able to update DataIds field — Nithish Albin
– Nithish Albin, Commented Oct 12, 2020 at 7:09
@Lupanoide Bulk updation is not possible with script. i don't whats wrong with the code. all fields inside "doc" is added as field in elastic search eg : "doc.script.lang :"painless" — Nithish Albin
– Nithish Albin, Commented Oct 12, 2020 at 7:41
Not sure if this is your issue but "_opt_type" should be "_op_type" — ekmcd
– ekmcd, Commented Oct 14, 2020 at 1:39

Val · Accepted Answer · 2020-10-19 04:30:13Z

7

+50

The correct way to upsert via script is without the doc but only the script section. You also need the upsert section if you want to upsert and update in the same command. It goes like this:

actions = [{
    "_op_type":"update",
    "_index": "test-data",
    "_type": "test-test-data",
    "_id": test_row ['testId'],
    "upsert": {
       "DataIds": test_row ['DataIds']
    },
    "script": {
        "source": "ctx._source.DataIds.add(params.DataIds)",
        "lang": "painless",
        "params": {
           "DataIds":test_row ['DataIds']
        }
    }
} for test_row in temp_list
]

Another way to do it is with scripted_upsert

actions = [{
    "_op_type":"update",
    "_index": "test-data",
    "_type": "test-test-data",
    "_id": test_row ['testId'],
    "upsert": {},
    "scripted_upsert": true,
    "script": {
        "source": "if (ctx._source.DataIds == null) { ctx._source.DataIds = []; } ctx._source.DataIds.add(params.DataIds)",
        "lang": "painless",
        "params": {
           "DataIds":test_row ['DataIds']
        }
    }
} for test_row in temp_list
]

answered Oct 19, 2020 at 4:30

Val

218k14 gold badges377 silver badges384 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

Nithish Albin Over a year ago

i am getting this error 'error': {'type': 'illegal_argument_exception', 'reason' : 'failed to execute script', 'caused_by': {'type': 'script_exception', 'reason': 'runtime error',@Val

Val Over a year ago

It's already better :-) Can you update your question with the full error you're getting?

Val Over a year ago

The error means that in some document dataIds is a string and not an array. Is that possible? dynamic method [java.lang.String, add/1] not found

Nithish Albin Over a year ago

yes it was because dataIds where string,it is fixed and when i tried with scripted_upsert way . Null values are getting appended in dataIds. may be because painless script

Val Over a year ago

I think you should create another question because this one is solved now and that's another issue.

|

Collectives™ on Stack Overflow

Elastic Search Bulk Update using python, how to append a array field with new data

1 Answer 1

10 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

10 Comments

Your Answer

Sign up or log in

Post as a guest

Related