Elasticsearch bulk/batch indexing with python requests module

Question

I have a smallish (~50,00) array of json dictionaries that I want to store/index in ES. My preference is to use python, since the data I want to index is coming from a csv file, loaded and converted to json via python. Alternatively, I would like to skip the step of converting to json, and simply use the array of python dictionaries I have. Anyway, a quick search revealed the bulk indexing functionality of ES. I want to do something like this:

post_url = 'http://localhost:9202/_bulk'
request.post(post_url, data = acc )    # acc a python array of dictionaries

or

post_url = 'http://localhost:9202/_bulk'
request.post(post_url, params = acc )    # acc a python array of dictionaries

both request give a [HTTP 500 error]

Rikard · Accepted Answer · 2014-10-31 07:24:13Z

4

My understanding is that you have to have one "command" per line (index, create, delete...) and then some of them (like index) takes a row of data on the next line like so

{'index': ''}\n
{'your': 'data'}\n
{'index': ''}\n
{'other': 'data'}\n

NB the new-lines, even on the last row.

Empty index objects like above works if you POST to ../index/type/_bulk or else you need to specify index and type I think, have not tried that.

answered Oct 31, 2014 at 7:24

Rikard

4094 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

alexander.polomodov · Accepted Answer · 2018-06-05 08:59:34Z

3

You the following function will do it:

def post_request(self, endpoint, data):
   endpoint = 'localhost:9200/_bulk'
   response = requests.post(endpoint, data=data, headers={'content-type':'application/json', 'charset':'UTF-8'})

   return response

As data you need to pass a String such:

{ "index" : { "_index" : "test-index", "_type" : "_doc", "_id" : "1681", "routing" : 0 }}
{ "field1" : ... , ..., "fieldN" : ... }
{ "index" : { "_index" : "test-index", "_type" : "_doc", "_id" : "1684", "routing" : 1 }}
{ "field1" : ... , ..., "fieldN" : ... }

Make sure you add a "\n" at the end of each line.

edited Jun 5, 2018 at 8:59

alexander.polomodov

5,54414 gold badges44 silver badges48 bronze badges

answered Jun 5, 2018 at 8:38

Alberto Castelo Becerra

4015 silver badges5 bronze badges

1 Comment

Pyd Over a year ago

what is the datatype of your data, shouldn't it be list of dicts?

dadoonet · Accepted Answer · 2012-12-22 07:49:37Z

0

I don't know much about Python, but did you look at Pyes? Bulk is supported in Pyes.

answered Dec 22, 2012 at 7:49

dadoonet

14.6k3 gold badges46 silver badges51 bronze badges

Collectives™ on Stack Overflow

Elasticsearch bulk/batch indexing with python requests module

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related