ElasticSearch filter by array item

Question

I have the following record in ES:

"authInput" : {
    "uID" : "foo",
    "userName" : "asdfasdfasdfasdf",
    "userType" : "External",
    "clientType" : "Unknown",
    "authType" : "Redemption_regular",
    "uIDExtensionFields" : 
    [
        {
            "key" : "IsAccountCreation",
            "value" : "true"
        }
    ],
    "externalReferences" : []
}

"uIDExtensionFields" is an array of key/value pairs. I want to query ES to find all records where:

"uIDExtensionFields.key" = "IsAccountCreation"
AND "uIDExtensionFields.value" = "true"

This is the filter that I think I should be using but it never returns any data.

GET devdev/authEvent/_search
{
   "size": 10,
    "filter": {
        "and": {
           "filters": [
              {
                  "term": {
                     "authInput.uIDExtensionFields.key" : "IsAccountCreation"
                  }
              },
              {
               "term": {
                  "authInput.uIDExtensionFields.value": "true"
               }   
              }
           ]
        }
    }
}

Any help you guys could give me would be much appreciated.

Cheers!

UPDATE: WITH THE HELP OF THE RESPONSES BELOW HERE IS HOW I SOLVED MY PROBLEM:

lowercase the value that I was searching for. (changed "IsAccoutCreation" to "isaccountcreation")
Updated the mapping so that "uIDExtensionFields" is a nested type
Updated my filter to the following:

_

GET devhilden/authEvent/_search
{
   "size": 10,
   "filter": {
      "nested": {
         "path": "authInput.uIDExtensionFields",
         "query": {
            "bool": {
               "must": [
                  {
                     "term": {
                        "authInput.uIDExtensionFields.key": "isaccountcreation"
                     }
                  },
                  {
                     "term": {
                        "authInput.uIDExtensionFields.value": "true"
                     }
                  }                  
               ]
            }
         }
      }
   }
}

try lower case key value of "isaccountcreation", depending on your mapping es might not care about case. — mconlin
– mconlin, Commented Feb 27, 2014 at 20:59
changing "IsAccountCreation" to "isaccountcreation" did indeed fix that part of my query, thank you very much! — jhilden
– jhilden, Commented Feb 27, 2014 at 21:57

Alex Brasetvik · Accepted Answer · 2014-02-27 21:09:32Z

5

There are a few things probably going wrong here.

First, as mconlin points out, you probably have a mapping with the standard analyzer for your key field. It'll lowercase the key. You probably want to specify "index": "not_analyzed" for the field.

Secondly, you'll have to use nested mappings for this document structure and specify the key and the value in a nested filter. That's because otherwise, you'll get a match for the following document:

"uIDExtensionFields" : [
    {
        "key" : "IsAccountCreation",
        "value" : "false"
    },
    {
        "key" : "SomeOtherField",
        "value" : "true"
    }
]

Thirdly, you'll want to be using the bool-filter's must and not and to ensure proper cachability.

Lastly, you'll want to put your filter in the filtered-query. The top-level filter is for when you want hits to be filtered, but facets/aggregations to not be. That's why it's renamed to post_filter in 1.0.

Here's a few resources you'll want to check out:

Troubleshooting Elasticsearch searches, for Beginners covers the first two issues.
Managing Relations in ElasticSearch covers nested docs (and parent/child)
all about elasticsearch filter bitsets covers and vs. bool.

answered Feb 27, 2014 at 21:09

Alex Brasetvik

11.8k2 gold badges40 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

jhilden Over a year ago

Thank you mconlin, you were correct that if I lower cased it then it worked when just searching on the key. @Alex, thank you for the suggestion about nesting, you are quite right. My question back to you: is that supported in ES 0.9x or do you need the 1.0 version to get that nesting? My updated query is working, but still has the nesting issue that needs to be fixed.

Collectives™ on Stack Overflow

ElasticSearch filter by array item

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related