Add Timestamp to ElasticSearch with Elasticsearch-py using Bulk-API

Question

I'm trying to add a timestamp to my data, have elasticsearch-py bulk index it, and then display the data with kibana.

My data is showing up in kibana, but my timestamp is not being used. When I go to the "Discovery" tab after configuring my index pattern, I get 0 results (yes, I tried adjusting the search time).

Here is what my bulk index json looks like:

{'index': 
         {'_timestamp': u'2015-08-11 14:18:26', 
          '_type': 'webapp_fingerprint', 
          '_id': u'webapp_id_redacted_2015_08_13_12_39_34',
          '_index': 'webapp_index'
         }
}

****JSON DATA HERE***

This will be accepted by elasticsearch and will get imported into Kibana, but the _timestamp field will not actually be indexed (it does show up in the dropdown when configuring an index pattern under "Time-field name").

I also tried formatting the metaFields like this:

{'index': {
           '_type': 'webapp_fingerprint', 
           '_id': u'webapp_id_redacted_2015_08_13_12_50_04', 
           '_index': 'webapp_index'
          }, 
           'source': {
                      '_timestamp': {
                                     'path': u'2015-08-11 14:18:26',
                                     'enabled': True, 
                                     'format': 'YYYY-MM-DD HH:mm:ss'
                                    }
                     }
}

This also doesn't work.

Finally, I tried including the _timestamp field within the index and applying the format, but I got an error with elasticsearch.

{'index': {
           '_timestamp': {
                          'path': u'2015-08-11 14:18:26', 
                          'enabled': True, 
                          'format': 'YYYY-MM-DD HH:mm:ss'
                         }, 
           '_type': 'webapp_fingerprint', 
           '_id': u'webapp_id_redacted_2015_08_13_12_55_53', 
           '_index': 'webapp_index'
          }
}

The error is:

elasticsearch.exceptions.TransportError: 
TransportError(500,u'IllegalArgumentException[Malformed action/metadata 
line [1], expected a simple value for field [_timestamp] but found [START_OBJECT]]')

Any help someone can provide would be greatly appreciated. I apologize if I haven't explained the issue well enough. Let me know if I need to clarify more. Thanks.

CookieEmpire · Accepted Answer · 2015-08-14 13:28:04Z

7

Fixed my own problem. Basically, I needed to add mappings for the timestamp when I created the index.

request_body = {
    "settings" : {
        "number_of_shards": 1,
        "number_of_replicas": 0
    },
    "mappings" : {
        "_default_":{
            "_timestamp":{
                 "enabled":"true",
                 "store":"true",
                 "path":"plugins.time_stamp.string",
                 "format":"yyyy-MM-dd HH:m:ss"
             }
         }
    }
}
print("creating '%s' index..." % (index_name))
res = es.indices.create(index = index_name, body = request_body)
print(" response: '%s'" % (res))

answered Aug 14, 2015 at 13:28

CookieEmpire

1791 gold badge1 silver badge9 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Sarindra Thérèse Over a year ago

Can you show me all of your code please?

Juanvulcano · Accepted Answer · 2020-05-05 15:18:00Z

2

In the latest versions of Elasticsearch, just using the PUT/POST API and ISOFORMAT strings should work.

import datetime
import requests 

query = json.dumps(
{
 "createdAt": datetime.datetime.now().replace(microsecond=0).isoformat(),
}
)
response = requests.post("https://search-XYZ.com/your-index/log", data=query,
               headers={'Content-Type': 'application/json'})
print(response)

answered May 5, 2020 at 15:18

Juanvulcano

1,3965 gold badges26 silver badges45 bronze badges

Collectives™ on Stack Overflow

Add Timestamp to ElasticSearch with Elasticsearch-py using Bulk-API

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related