Elasticsearch lowercase filter search

Question

I'm trying to search my database and be able to use upper/lower case filter terms but I've noticed while query's apply analyzers, I can't figure out how to apply a lowercase analyzer on a filtered search. Here's the query:

{
    "query": {
        "filtered": {
            "filter": {
                "bool": {
                    "should": [
                        {
                            "term": {
                                "language": "mandarin" // Returns a doc
                            }
                        },
                        {
                            "term": {
                                "language": "Italian" // Does NOT return a doc, but will if lowercased
                            }
                        }
                    ]
                }
            }
        }
    }
}

I have a type languages that I have lowercased using:

"analyzer": {
    "lower_keyword": {
        "type": "custom",
        "tokenizer": "keyword",
        "filter": "lowercase"
    }
}

and a corresponding mapping:

"mappings": {
    "languages": {
        "_id": {
            "path": "languageID"
        },
        "properties": {
            "languageID": {
                "type": "integer"
            },
            "language": {
                "type": "string",
                "analyzer": "lower_keyword"
            },
            "native": {
                "type": "string",
                "analyzer": "keyword"
            },
            "meta": {
                "type": "nested"
            },
            "language_suggest": {
                "type": "completion"
            }
        }
    }
}

John Petrone · Accepted Answer · 2014-07-17 19:36:56Z

9

The problem is that you have a field that you have analyzed during index to lowercase it, but you are using a term filter for the query which is not analyzed:

Term Filter

Filters documents that have fields that contain a term (not analyzed). Similar to term query, except that it acts as a filter.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-term-filter.html

I'd try using a query filter instead:

Query Filter

Wraps any query to be used as a filter. Can be placed within queries that accept a filter.

Example:

{
    "constantScore" : {
        "filter" : {
            "query" : {
                "query_string" : {
                    "query" : "this AND that OR thus"
                }
            }
        }
    } }

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-query-filter.html#query-dsl-query-filter

answered Jul 17, 2014 at 19:36

John Petrone

27.5k6 gold badges68 silver badges69 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Maruf Over a year ago

So if I wanted tho term to be lowercased would I change the tokenizer to a lowercase one and reindex everything?

John Petrone Over a year ago

No, it's already lowercased during indexing due to the lowercase filter. The problem is you need to use a query type that will also analyze - Term filters do not.

odyth Over a year ago

Would you see better performance if you just lowercased your input so you could continue using a Term Filter over using a Query Filter?

Ariel C. · Accepted Answer · 2020-04-01 14:33:27Z

This may be achieved by appending .keyword to your field to query against the keyword version of the field. Assuming language was defined in the mapping with type keyword.

Note that now only the exact text would match: mandarin won't match and Italian would.

Your query would end up like this:

{
    "query": {
        "filtered": {
            "filter": {
                "bool": {
                    "should": [
                        {
                            "term": {
                                "language.keyword": "mandarin" // Returns Empty
                            }
                        },
                        {
                            "term": {
                                "language.keyword": "Italian" // Returns Italian.
                            }
                        }
                    ]
                }
            }
        }
    }
}

Combining the term values is also allowed:

{
    "query": {
        "filtered": {
            "filter": {
                "bool": {
                    "should": [
                        {
                            "term": {
                                "language.keyword":
                                     ["mandarin", "Italian"]
                            }
                        }
                    ]
                }
            }
        }
    }
}

Collectives™ on Stack Overflow

Elasticsearch lowercase filter search

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related