Template file not working in ElasticSearch

Question

I have an external JSON template file to load into ElasticSearch

This is what I do:

curl -XPUT 'http://localhost:9200/_template/mytemplate' -d @file.json

The command get correctly acknowledged

Unfortunately when the index is created the rules defined inside my JSON file are not applied

EDIT

This is the JSON file

{
    "template" : "log-*",
    "settings": {
        "index": {
            "number_of_shards": 1,
            "number_of_replicas": 0
        }
    },
    "mappings": {
        "logEvent": {
            "properties": {
                "timeStamp": {
                    "type": "date",
                    "format": "dateOptionalTime"
                },
                "message": {
                    "type": "string"
                },
                "messageObject": {
                    "type": "object"
                },
                "exception": {
                    "type": "object"
                },
                "loggerName": {
                    "type": "string"
                },
                "domain": {
                    "type": "string"
                },
                "identity": {
                    "type": "string"
                },
                "level": {
                    "type": "string"
                },
                "className": {
                    "type": "string"
                },
                "fileName": {
                    "type": "string"
                },
                "lineNumber": {
                    "type": "long"
                },
                "fullInfo": {
                    "type": "string"
                },
                "methodName": {
                    "type": "string"
                },
                "fix": {
                    "type": "string"
                },
                "userName": {
                    "type": "string"
                },
                "threadName": {
                    "type": "string"
                },
                "hostName": {
                    "type": "string"
                }
            }
        }
    }
}

which should be applied to any index matching log-*. One of those index is log-2016.07.28

The template specifies the type of lineNumber. It should change the type of such lineNumber field from the default string to long. What I get is a document with lineNumber as a string.

This is the returned document:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "log-2016.07.28",
      "_type" : "logEvent",
      "_id" : "AVYwvw-k6GHUP7T-sYlL",
      "_score" : 1.0,
      "_source" : {
        "timeStamp" : "2016-07-28T09:04:02.8994786Z",
        "message" : "Upload file operation took 600 ms",
        "messageObject" : { },
        "exception" : { },
        "loggerName" : "Reviewer.Web.WebApi.GroupsController",
        "domain" : "/LM/W3SVC/2/ROOT-1-131141667495593380",
        "identity" : "",
        "level" : "INFO",
        "className" : "Reviewer.Logger.MethodTimer",
        "fileName" : "MethodTimer.cs",
        "lineNumber" : "49",
        "fullInfo" : "MethodTimer.cs:49)",
        "methodName" : "Dispose",
        "fix" : "LocationInfo, UserName, Identity, Partial",
        "properties" : {
          "test" : "123",
          "log4net:HostName" : "GBWOTIOM68052D",
          "IP" : "::1",
          "log4net:Identity" : "",
          "log4net:UserName" : "CORP\\gianluca.ghettini",
          "log4net:ElapsedTime" : "600",
          "@timestamp" : "2016-07-28T09:04:02.8994786Z"
        },
        "userName" : "CORP\\gianluca.ghettini",
        "threadName" : "198",
        "hostName" : "GBWOTIOM68052D"
      }
    } ]
  }
}

as you can see the

"lineNumber" : "49",

is still a string instead of a long

Can you show how you index your log documents? Please show one sample document. — Val
– Val, Commented Jul 28, 2016 at 9:17
I just added a sample document, what I'd like to have and what I actually get instead — Gianluca Ghettini
– Gianluca Ghettini, Commented Jul 28, 2016 at 9:23

Val · Accepted Answer · 2016-07-28 12:45:05Z

3

What you observe is the source of the document (as it was sent to ES) and ES will never change it. If your source contains a string value, you'll see a string value, if your source contains a numeric value, you'll see a number value in the source.

However, the way the data is indexed is what really matters. If your mapping declares a given field to be a string, the field value in the source (be it a number, a boolean, a string or whatever) will be indexed as a string.

If your mapping declares a given field to be a number and the field value in the source is a string, ES will try to coerce the string into a number and that number will be indexed, however, the string in the source will not be changed to a number.

So, in your case, you send lineNumber as the string "49", so ES will coerce the string "49" to the number 49 and index that number, even though, the source will still contain the string "49".

To sum it up, if you really want to see a number in your source, you need to send a number, i.e. "lineNumber": 49 instead of "lineNumber": "49"

edited Jul 28, 2016 at 12:45

answered Jul 28, 2016 at 12:35

Val

218k14 gold badges377 silver badges384 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Gianluca Ghettini Over a year ago

brilliant! Is there a way to know if I got the "right" type? Is it possible to query ES for such info?

Gianluca Ghettini Over a year ago

I mean, I way to actually verify the field has got the right type internally

Val Over a year ago

Trust your mapping and ES for doing its job ;)

Gianluca Ghettini Over a year ago

of course. My only concern is that the field I'm trying to change is not actually "lineNumber" but instead "log4net:ElapsedTime" which is inside the "properties" object, an unfortunate name for a field ("properties" is also the JSON keyword used by ES to define the field list)

Val Over a year ago

You can definitely modify your mapping to include the properties object and the mappings of its inner fields.

Collectives™ on Stack Overflow

Template file not working in ElasticSearch

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related