Python requests.post does not force Elasticsearch to create missing index

Question

I want to push data to my Elasticsearch server using :

requests.post('http://localhost:9200/_bulk', data=data_1 + data_2)

and it complains that the index does not exist. I try creating the index manually:

    curl -X PUT http://localhost:9200/_bulk

and it complains that I am not feeding a body to it:

    {"error":{"root_cause":[{"type":"parse_exception","reason":"request body is required"}],"type":"parse_exception","reason":"request body is required"},"status":400}

Seems like a bit of a chicken and egg problem here. How can I create that _bulk index, and then post my data?

EDIT:

My data is very large to even understand the schema. Here is a small snippet:

'{"create":{"_index":"products-guys","_type":"t","_id":"0"}}\n{"url":"http://www.plaisio.gr/thleoraseis/tv/tileoraseis/LG-TV-43-43LH630V.htm","title":"TV LG 43\\" 43LH630V LED Full HD Smart","description":"\\u039a\\u03b1\\u03b9 \\u03cc\\u03bc\\u03bf\\u03c1\\u03c6\\u03b7 \\u03ba\\u03b1\\u03b9 \\u03ad\\u03be\\u03c5\\u03c0\\u03bd\\u03b7, \\u03bc\\u03b5 \\u03b9\\u03c3\\u03c7\\u03c5\\u03c1\\u03cc \\u03b5\\u03c0\\u03b5\\u03be\\u03b5\\u03c1\\u03b3\\u03b1\\u03c3\\u03c4\\u03ae \\u03b5\\u03b9\\u03ba\\u03cc\\u03bd\\u03b1\\u03c2 \\u03ba\\u03b1\\u03b9 \\u03bb\\u03b5\\u03b9\\u03c4\\u03bf\\u03c5\\u03c1\\u03b3\\u03b9\\u03ba\\u03cc webOS 3.0 \\u03b5\\u03af\\u03bd\\u03b1\\u03b9 \\u03b7 \\u03c4\\u03b7\\u03bb\\u03b5\\u03cc\\u03c1\\u03b1\\u03c3\\u03b7 \\u03c0\\u03bf\\u03c5 \\u03c0\\u03ac\\u03b5\\u03b9 \\u03c3\\u03c4\\u03bf \\u03c3\\u03b1\\u03bb\\u03cc\\u03bd\\u03b9 \\u03c3\\u03bf\\u03c5","priceCurrency":"EUR","price":369.0}\n{"create":{"_index":"products-guys","_type":"t","_id":"1"}}\n{"url":"http://www.plaisio.gr/thleoraseis/tv/tileoraseis/Samsung-TV-43-UE43M5502.htm","title":"TV Samsung 43\\" UE43M5502 LED ...

This is essentially someone else's code, that I need to make work. It seems that the "data" object I am passing to the PUT method is a string.

When I use requests.post('http://localhost:9200/_bulk', data=data)

I get <Response [406]>,

llermaly · Accepted Answer · 2021-03-10 13:42:52Z

1

If you want to do a bulk request using requests

response = requests.post('http://localhost:9200/_bulk', data= data=data_1 + data_2, headers={'content-type':'application/json', 'charset':'UTF-8'})

Old Answer

I recommend using the bulk helper from the python library

from elasticsearch import Elasticsearch, helpers

client = Elasticsearch("localhost:9200")

def gendata():
    mywords = ['foo', 'bar', 'baz']
    for word in mywords:
        yield {
            "_index": "mywords",
            "word": word,
        }

resp = helpers.bulk(
client,
gendata(),
index = "some_index",
)

If you didn't touch the elasticsearch configuration a new index will be created on document indexing.

About the ways you tried:

Probably the query is malformed. To do bulk ingest the body shape is different than just sending the docs as a json array.
You are doing PUT instead of post and you have to specify the documents you want to ingest.

There is no need to create the empty index first. Just in case you want to do you can just do :

curl -X PUT http://localhost:9200/index_name

edited Mar 10, 2021 at 13:42

answered Mar 10, 2021 at 12:34

llermaly

2,5283 gold badges20 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

11 Comments

Qubix Over a year ago

Not sure how the above helps. All I want is to replace the line requests.post('localhost:9200/_bulk', data=data_1 + data_2) I already have the data.

llermaly Over a year ago

Can I see what data are you pushing? if you are not specifying the index name in the body then you have localhost:9200/index_name/_bulk Your body should look like { "index" : { "_index" : "test", "_id" : "1" } } { "field1" : "value1" }

Qubix Over a year ago

unfortunately I cannot share the data. Basically all I want is just to create that _bulk index and add the data in json format to it.

llermaly Over a year ago

But what's the shape of the data? if is an array of json it will not work without the helper I showed you. If you want to use POST directly you have to parse into the right format elastic.co/guide/en/elasticsearch/reference/current/…

llermaly Over a year ago

try adding this Content-Type to your request application/json; charset=utf-8

|

Collectives™ on Stack Overflow

Python requests.post does not force Elasticsearch to create missing index

1 Answer 1

11 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

11 Comments

Your Answer

Sign up or log in

Post as a guest

Related