How to get Elasticsearch terms aggregation for multi valued fields using NGram filter for autocompletion?

Question

I am doing my autocompletion project and new to Elasticsearch. I have used Edge NGram filter for autocompletion. I am trying to get unique results for autocompletion, So I have used terms aggregation for all the fields. I am getting good results for the field having 1 value, but for the fields having more than one value.. if the query matches at least one value from that field.. it is giving me all the values from that field(whether the query matches or not in other values).

My settings and mapping under the garments index are :

PUT /garments
{
  "settings" : 
  {
    "number_of_replicas": 3,
    "number_of_shards": 2,
    "analysis": 
    {
      "analyzer": 
      {
        "autocomplete": 
        {
          "tokenizer": "autocomplete",
          "filter": 
          [
            "lowercase"
          ]
        },
        "autocomplete_search": 
        {
          "tokenizer": "lowercase"
        }
      },
      "tokenizer": 
      {
        "autocomplete": 
        {
          "type": "edge_ngram",
          "min_gram": 2,
          "max_gram": 10,
          "token_chars": 
          [
            "letter"
          ]
        }
      }

    }
  },
  "mappings":
  {
    "properties":
    {
      "color": 
      {
        "type": "text",
        "analyzer": "autocomplete",
        "search_analyzer": "autocomplete_search",
        "fields": 
        {
          "keyword": 
          { 
            "type": "keyword"
          }
        }

      }
........
........
........
   }
}

(note that i am using text type) suppose I have a color field in a doc having multi values like :["blue","black","orange","marble","jet black"] and my search query is :

GET /garments/_search
{
  "size": 0, 
  "query": 
  {
      "query_string": {
        "query": "bl"
      }
  },
  "aggs":
  {
    "Term_aggregation": 
    {
      "terms": 
      {
        "field": "color.keyword", 
        "size": 100
      }
    }
  }
}

this gives me all the outputs i.e.: "blue","black","orange","marble","jet black". But i wanted only blue, black, jet black as my results(query is "bl"). Later I used

"include": " .*bl.*"

filter in my terms aggs.. which gave me blue,black,marble,jet black as my results.. this include filter is case sensitive... Please help!

You want a case insensitive wildcard search in "include"?

jaspreet chahal
– jaspreet chahal

2020-05-22 08:22:54 +00:00
Commented May 22, 2020 at 8:22 — jaspreet chahal
– jaspreet chahal, Commented May 22, 2020 at 8:22

jaspreet chahal · Accepted Answer · 2020-05-22 11:24:41Z

0

If you want to do case insensitive match on a keyword field, you can use normalizer with a lowercase filter

The normalizer property of keyword fields is similar to analyzer except that it guarantees that the analysis chain produces a single token.

{
  "settings": {
    "analysis": {
      "normalizer": {
        "lowercase_normalizer": {
          "type": "custom",
          "filter": [
            "lowercase"
          ]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "color": {
        "type": "text",
        "analyzer": "autocomplete",
        "search_analyzer": "autocomplete_search",
        "fields": {
          "keyword": {
            "type": "keyword",
            "normalizer": "lowercase_normalizer"
          }
        }
      }
    }
  }
}

"include": " .bl." will work even if actual value has uppercase leters

EDIT 1

As per your comment if you don't want to use include in terms. You need to index your color with nested type, so that each color is treated as separate object

Mappings:

PUT index64
{
  "settings": {
    "number_of_replicas": 3,
    "number_of_shards": 2,
    "analysis": {
      "analyzer": {
        "autocomplete": {
          "tokenizer": "autocomplete",
          "filter": [
            "lowercase"
          ]
        },
        "autocomplete_search": {
          "tokenizer": "lowercase"
        }
      },
      "tokenizer": {
        "autocomplete": {
          "type": "edge_ngram",
          "min_gram": 2,
          "max_gram": 10,
          "token_chars": [
            "letter"
          ]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "color": {
        "type": "nested",
        "properties": {
          "name": {
            "type": "text",
            "analyzer": "autocomplete",
            "search_analyzer": "autocomplete_search",
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          }
        }
      }
    }
  }
}

Query:

POST index64/_doc
{
  "color": [
    {
      "name": "blue"
    },
    {
      "name": "black"
    },
    {
      "name": "orange"
    },
    {
      "name": "marble"
    },
    {
      "name": "jet black"
    }
  ]
}

Result:

GET index64/_search
{
  "size": 0, 
  "aggs": {
    "color": {
      "nested": {
        "path": "color"
      },
      "aggs": {
        "select_color": {
          "filter": {
            "match":{
              "color.name":"bl"
            }
          },
          "aggs": {
            "distinct_colors": {
              "terms": {
                "field": "color.name.keyword",
                "size": 10
              }
            }
          }
        }
      }
    }
  }
}

Result

"aggregations" : {
    "color" : {
      "doc_count" : 5,
      "select_color" : {
        "doc_count" : 3,
        "distinct_colors" : {
          "doc_count_error_upper_bound" : 0,
          "sum_other_doc_count" : 0,
          "buckets" : [
            {
              "key" : "black",
              "doc_count" : 1
            },
            {
              "key" : "blue",
              "doc_count" : 1
            },
            {
              "key" : "jet black",
              "doc_count" : 1
            }
          ]
        }
      }
    }
  }

edited May 22, 2020 at 11:24

answered May 22, 2020 at 8:32

jaspreet chahal

9,1392 gold badges14 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Saurav Sharma Over a year ago

Hello, @jaspreet..Thanks for your solution,but is there any other way in which I don't have to use include field.. rather when I search for a query "bl".. it should give black,blue,jet black as the results in term aggregation.

Saurav Sharma Over a year ago

Hello @jaspreet, this approach was really helpful, but instead of using the filter in the aggs (while calling _search API in your code)... can I use a simple search query (as I have used above in my code.. before writing aggs) for searching the color names starting with "bl" ?

jaspreet chahal Over a year ago

@SauravSharma you can filter in the query. If you're are using nested field, you will need to use nested query in query part

Collectives™ on Stack Overflow

How to get Elasticsearch terms aggregation for multi valued fields using NGram filter for autocompletion?

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related