Select documents by array of objects when at least one object doesn't contain necessary field Elasticsearch

Question

I have documents in the elasticsearch and can't understand how to apply search script that should return documents if any attachment doesn't contain uuid or uuid is null. Version of elastic 5.2. Mapping of documents

"mappings": {
    "documentType": {
        "properties": {
            "attachment": {
                "properties": {
                    "uuid": {
                        "type": "text"
                    },
                    "path": {
                        "type": "text"
                    },
                    "size": {
                        "type": "long"
                    }
                }
            }}}

In the elasticsearch it looks like

{
        "_index": "documents",
        "_type": "documentType",
        "_id": "1",
        "_score": 1.0,
        "_source": {
          "attachment": [
               {
                "uuid": "21321321",
                "path": "../uploads/somepath",
                "size":1231
               },
               {
                "path": "../uploads/somepath",
                "size":1231
               },      
         ]},
{
        "_index": "documents",
        "_type": "documentType",
        "_id": "2",
        "_score": 1.0,
        "_source": {
          "attachment": [
               {
                "uuid": "223645641321321",
                "path": "../uploads/somepath",
                "size":1231
               },
               {
                "uuid": "22341424321321",
                "path": "../uploads/somepath",
                "size":1231
               },        
         ]},
{
        "_index": "documents",
        "_type": "documentType",
        "_id": "3",
        "_score": 1.0,
        "_source": {
          "attachment": [
               {
                "uuid": "22789789341321321",
                "path": "../uploads/somepath",
                "size":1231
               }, 
               {
                "path": "../uploads/somepath",
                "size":1231
               },      
         ]}

As result I want to get attachments with _id 1 and 3. But as result I get error of the script I tried to apply next script:

{
    "query": {
        "bool": {
            "must": [
                {
                    "exists": {
                        "field": "attachment"
                    }
                },
                {
                    "script": {
                        "script": {
                            "inline": "for (item in doc['attachment'].value) { if (item['uuid'] == null) { return true}}",
                            "lang": "painless"
                        }
                    }
                }
            ]
        }
    }
}

Error is next:

 "root_cause": [
            {
                "type": "script_exception",
                "reason": "runtime error",
                "script_stack": [
                    "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:77)",
                    "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:36)",
                    "for (item in doc['attachment'].value) { ",
                    "                 ^---- HERE"
                ],
                "script": "for (item in doc['attachment'].value) { if (item['uuid'] == null) { return true}}",
                "lang": "painless"
            }
        ],

Is it possible to select documents in case even one attachment object doesn't contain uuid ?

Jozef - Spatialized.io · Accepted Answer · 2021-04-14 08:29:47Z

2

Iterating arrays of objects is not as trivial as one would expect. I've written extensively about it here and here.

Since your attachments are not defined as nested, ES will internally represent them as flattened lists of values (also called "doc values"). For instance attachment.uuid in doc#2 will become ["223645641321321", "22341424321321"], and attachments.size will turn into [1231, 1231].

This means that you can simply compare the .length of these flattened representations! I assume attachment.size will always be present and can be thus taken as the comparison baseline.

One more thing. To take advantage of these optimized doc values for textual fields, it'll require one small mapping change:

PUT documents/documentType/_mappings
{
  "properties": {
    "attachment": {
      "properties": {
        "uuid": {
          "type": "text",
          "fielddata": true     <---
        },
        "path": {
          "type": "text"
        },
        "size": {
          "type": "long"
        }
      }
    }
  }
}

When that's done and you've reindexed your docs — which can be done with this little Update by query trick:

POST documents/_update_by_query

You can then use the following script query:

POST documents/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "exists": {
            "field": "attachment"
          }
        },
        {
          "script": {
            "script": {
              "inline": "def size_field_length = doc['attachment.size'].length; def uuid_field_length =  doc['attachment.uuid'].length; return uuid_field_length < size_field_length",
              "lang": "painless"
            }
          }
        }
      ]
    }
  }
}

answered Apr 14, 2021 at 8:29

Jozef - Spatialized.io

17k4 gold badges29 silver badges79 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Stepan Poperechnyi Over a year ago

thank you for your response. I will check it and looks for future I will make reindex for attachment that change type on nested.

Jozef - Spatialized.io Over a year ago

You're welcome. Making the attachments nested is a good approach. Just keep in mind that scripting with nested fields is a chapter of its own. The links at the top of my answer should point you to the right direction, though!

Stepan Poperechnyi Over a year ago

It works. I have reindexed and run your script and it works. Thank you a lot.

Nikita Borisov · Accepted Answer · 2021-04-15 10:21:58Z

1

Just to supplement this answer. If mapping for uuid field was created automatically elastic search adds it in this way:

"uuid": {
    "type": "text",
    "fields": {
        "keyword": {
            "type": "keyword",
            "ignore_above": 256
        }
    }
}

then script could look like:

POST documents/_search
{
    "query": {
        "bool": {
            "must": [
                {
                    "exists": {
                        "field": "attachment"
                    }
                },
                {
                    "script": {
                        "script": {
                            "inline": "doc['attachment.size'].length > doc['attachment.uuid.keyword'].length",
                            "lang": "painless"
                        }
                    }
                }
            ]
        }
    }
}

answered Apr 15, 2021 at 10:21

Nikita Borisov

213 bronze badges

1 Comment

Stepan Poperechnyi Over a year ago

you are right, your suggestion also works fine even without reindexing

Collectives™ on Stack Overflow

Select documents by array of objects when at least one object doesn't contain necessary field Elasticsearch

2 Answers 2

3 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related