update mapping property for data going to elasticsearch index

Question

New to elasticsearch here and debugging an existing index. I was creating a dashboard based on a search and found that some fields that were being sent do not appear as an option to filter on. I checked further into this and saw that there are some fields that are not indexed. The person who created the index claims that there is no restriction on what fields are being indexed but I disagree having found the following:

     "customerid": {
              "type": "string",
              "norms": {
                 "enabled": false
              },
              "fields": {
                 "raw": {
                    "type": "string",
                    "index": "not_analyzed",
                    "ignore_above": 256
                 }
              }
           },

It shows the value as not_analyzed. I would like to update this value so that the fields I need are indexed and available for visualizations in the dashboard. I know the index name but the data itself and mapping is under types. So not sure how to do this. When looking in elasticsearch plugin/head I do not see the index.

"customer_index": {
        "dynamic_templates": [
           {
              "string_fields": {
                 "mapping": {
                    "index": "analyzed",
                    "omit_norms": true,
                    "type": "multi_field",
                    "fields": {

except this initial headers for all fields related. Any suggestions or help will be appreciated.

EDIT:

As pointed out correctly by Alain, my misunderstanding of not_analyzed. I am still confused a bit and hope that adding some additional information will help diagnose this problem.

Firstly, this is a view of kibana that shows for the specific index that contains the data, the available fields:

Available Fields
   @timestamp
   _id
   _type
   etc.

Customer ID is not one of them. Now there are different data sources coming to the same index for example :

job records
customer records
project records

etc.

This is defined by _type field. Now I want to access the customer record object and it has its own properties:

customer_index": {
        "dynamic_templates": [
           {
              "string_fields": {
                 "mapping": {
                    "index": "analyzed",
                    "omit_norms": true,
                    "type": "multi_field",
                    "fields": {
                       "{name}": {
                          "index": "analyzed",
                          "omit_norms": true,
                          "type": "string"
                       },
                       "raw": {
                          "ignore_above": 256,
                          "index": "not_analyzed",
                          "type": "string"
                       }
                    }
                 },
                 "match": "*",
                 "match_mapping_type": "string"
              }
           },
           {
              "message_field": {
                 "mapping": {
                    "index": "analyzed",
                    "omit_norms": true,
                    "type": "string"
                 },
                 "match": "message",
                 "match_mapping_type": "string"
              }
           }
        ],
        "_all": {
           "enabled": true,
           "omit_norms": true
        },
        "properties": {
           "@timestamp": {
              "type": "date",
              "format": "dateOptionalTime"
           },
           "@version": {
              "type": "string",
              "index": "not_analyzed"
           },
           "CCType": {
              "type": "string",
              "norms": {
                 "enabled": false
              },
              "fields": {
                 "raw": {
                    "type": "string",
                    "index": "not_analyzed",
                    "ignore_above": 256
                 }
              }
           },
           "Crawled": {
              "type": "string",
              "norms": {
                 "enabled": false
              },
              "fields": {
                 "raw": {
                    "type": "string",
                    "index": "not_analyzed",
                    "ignore_above": 256
                 }
              }
           },
           "customerid": {
              "type": "string",
              "norms": {
                 "enabled": false
              },
              "fields": {
                 "raw": {
                    "type": "string",
                    "index": "not_analyzed",
                    "ignore_above": 256
                 }
              }
           },

Now I would like to search for these property fields. I found SENSE plugin which I am trying to use to learn the queries and was able to do this:

GET _search
{
"query": {
    "match": {
       "customerid": "11908906"
    }
}

}

This worked great in returning the messages and counts. Now when I try within Kibana in discover tab, I search for the type (customer_index) and filter the field I need and I see all the results. I just wondering how to translate this visually.

Alain Collins · Accepted Answer · 2015-09-16 21:15:16Z

3

"not_analyzed" does not mean "not indexed". It means that elasticsearch is not trying to analyze the string, instead leaving it intact. This is a very common thing in elasticsearch, depending on your content.

For example:

/var/log/messages

will, by default, be split into:

[ "var", "log", "messages" ]

which is not very useful when you want to search on it as a full path. By setting the field to not_analyzed, it will leave it alone. You can still search using the field.

Second, you say that the fields aren't available for filtering. Assuming you're using Kibana, be aware that Kibana will cache the field mapping, meaning that new fields don't automatically show up. Go to Settings->Indices, select your index, and click Reload Field List.

answered Sep 16, 2015 at 21:15

Alain Collins

16.4k2 gold badges34 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

vbNewbie Over a year ago

Thanks. I am clear on the meaning of 'not_analyzed', but after refreshing the index, I still do not see the fields there. I will try and add some more information

Alain Collins Over a year ago

Then what does "[I] saw that there are some fields that are not indexed" mean, when the fields are actually not_analyzed?

vbNewbie Over a year ago

That was my misunderstanding as you correctly pointed out. What I am unsure of is how to query the field. the field may be not_analyzed as you said but when I created a pie graph and aggregated on terms, then tried to find the term, it was not there. I think I need to create a specific query that will find that field. I will update my question above accordingly

Alain Collins Over a year ago

In Kibana, you can search for "customerId: 11908906" and it should work like the DSL example you provided.

vbNewbie Over a year ago

Yes. But if I need to add a pie chart and base it on a saved search for all customer ids where I can display a count for each id, I need to aggregate by term or filter and the option for terms does not have customerid. When I base it on filter customerid:"*" I get 0 results even when a histogram shows incoming data all the time

Collectives™ on Stack Overflow

update mapping property for data going to elasticsearch index

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related