Bulk Partial Upsert in Elasticseach with python

Question

I want to send n upsert partial requests to ES, is such a thing possible? So if the document doesn't exist, insert my partial doc. If it already exists, update it with the partial doc.

Using the bulk helpers, I've tried a ton of variations, but they all wipe out existing values in favour of the new ones.

data = [{
    "_index": 'my_index',
    "_type": 'my_type',
    "_id": 12345,
    "doc": {"newkey": 'newvalue'}
}]
helpers.bulk(es, data, index='my_index', doc_type='my_type')

or

data = [{
    "_index": 'my_index',
    "_type": 'my_type',
    "_id": 12345,
    "_source": {"newkey": 'newvalue'}
}]
helpers.bulk(es, data, index='my_index', doc_type='my_type')

Does not work either.

jhandei · Accepted Answer · 2017-03-18 14:44:51Z

17

As J. Ku answered, he has given the right answer but the code provided is incomplete. So posting the complete answer.

data = [{
    "_op_type": 'update',
    "_index": 'my_index',
    "_type": 'my_type',
    "_id": 12345,
    "doc": {"newkey": 'newvalue'},
    "doc_as_upsert":True
}]

helpers.bulk(es, data, index='my_index', doc_type='my_type')

answered Mar 18, 2017 at 14:44

jhandei

1831 silver badge12 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Kailegh Over a year ago

what if you dont know the _id?

jhandei Over a year ago

Haven't worked on elastic search for a long time. However I feel, _id is necessary in upsert, else you cannot identify whether the doc is there or not. And if you don't know it, you have to make another query to fetch all the ids you want to update.

J.Ku · Accepted Answer · 2015-12-23 20:19:24Z

1

I think you need to include the action as mentioned in the documentation and set upsert as true

data = [{
    "_op_type": 'update',
    "_index": 'my_index',
    "_type": 'my_type',
    "_id": 12345,
    "doc": {"newkey": 'newvalue'}
}]
helpers.bulk(es, data, index='my_index', doc_type='my_type')

edited Dec 23, 2015 at 20:19

answered Dec 23, 2015 at 19:48

J.Ku

1237 bronze badges

4 Comments

L-R Over a year ago

doc_as_upsert does not make a difference. Also I want the action to be index, which is default if you leave it out. Finally, I'm using the bulk API which is different syntax from what you posted.

J.Ku Over a year ago

I believe the index action always overwrites the existing value if exist.

J.Ku Over a year ago

not sure how to do it with index. my answer uses update and it would behave like you expected

L-R Over a year ago

sorry, I had the syntax wrong, I think the above works.

Collectives™ on Stack Overflow

Bulk Partial Upsert in Elasticseach with python

2 Answers 2

2 Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related