0

Background: We have ~140 million polygons split into 5 indices (region-[1-5]) with 2 shards each. It was loaded with ES 7.10. The field containing the polygon is named 'shape' and is mapped as a geo_shape field.

Here's an indexed example:

"shape": {
  "type": "Polygon",
  "coordinates": [
    [
      [
        -80.661103428642,
        28.0213473946004
      ],
      [
        -80.6611091545036,
        28.0210035893407
      ],
      [
        -80.6615120749597,
        28.021009053184
      ],
      [
        -80.6615063490981,
        28.0213528568402
      ],
      [
        -80.661103428642,
        28.0213473946004
      ]
    ]
  ]
},

Our problem occurs when querying for polygons which intersect a given (usually hand-drawn) shape. e.g.:

GET region_parcels*/_search
{
  "query": {
    "bool": {
      "filter": {
        "geo_shape": {
          "shape": {
            "shape": {
              "type": "POLYGON",
              "coordinates": [
                [
                  [
                    -81.0864386380646,
                    32.07339101099513
                  ],
                  [
                    -81.0890350163911,
                    32.07282734995984
                  ],
                  [
                    -81.08907793173533,
                    32.07190002908301
                  ],
                  [
                    -81.08796213278512,
                    32.07151818834138
                  ],
                  [
                    -81.08648155340886,
                    32.071481822473295
                  ],
                  [
                    -81.08459327826233,
                    32.07231823378
                  ],
                  [
                    -81.0841426671478,
                    32.073136454828834
                  ],
                  [
                    -81.08480785498352,
                    32.073645566452704
                  ],
                  [
                    -81.08527992377016,
                    32.07390012120158
                  ],
                  [
                    -81.08530138144226,
                    32.07390012120158
                  ],
                  [
                    -81.0864386380646,
                    32.07339101099513
                  ]
                ]
              ]
            },
            "relation": "intersects"
          }
        }
      }
    }
  },
  "size": 1000
}

When we run the above query, we are getting some results that are up to 30ft outside of the drawn polygon. The false positives are not uniform (we can't just negative buffer our search polygon to return correct intersections). We have also dropped a single point as the search geometry in the middle of 1 of our indexed polygons and have gotten the intersected polygon as well as a few of the surrounding polygons back.

Reading over the docs and blogs, it looks like specifying any sort of precision is still available but will soon be deprecated and that the new tessellation technique for indexing is supposed to be accurate up to a few mm out of the box.

Is there any way to set up the index/cluster or execute the query differently that we have overlooked to make spatial intersection queries more accurate?

Thank you.

Edit

Here is an actual example with a point in the center of one of the polygons. It returns 3 hits, the intersected one (correct) and one from either side of the intersected (incorrect).: Request:

GET region_parcels*/_search
{
  "query": {
    "bool": {
      "filter": {
        "geo_shape": {
          "shape": {
            "shape": {
              "type": "POINT",
              "coordinates": [
                -81.08111523359743,
                32.04772418111284
              ]
            },
            "relation": "intersects"
          }
        }
      }
    }
  },
  "_source": ["shape"],
  "explain": true,
  "size": 1000
}

Response:

{
  "took" : 8,
  "timed_out" : false,
  "_shards" : {
    "total" : 10,
    "successful" : 10,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : 0.0,
    "hits" : [
      {
        "_shard" : "[<my_index>][0]",
        "_node" : "lrSfQEyVTWmWU828O6Qdsw",
        "_index" : "<my_index>",
        "_type" : "_doc",
        "_id" : "cY2O9XcBlBVQyhnplhLN",
        "_score" : 0.0,
        "_source" : {
          "shape" : {
            "coordinates" : [
              [
                [
                  -81.0810247260436,
                  32.0478338967803
                ],
                [
                  -81.0811253535251,
                  32.0475727349866
                ],
                [
                  -81.0812173428069,
                  32.0475984458201
                ],
                [
                  -81.0811167162237,
                  32.0478596090633
                ],
                [
                  -81.0810247260436,
                  32.0478338967803
                ]
              ]
            ],
            "type" : "Polygon"
          }
        },
        "_explanation" : {
          "value" : 0.0,
          "description" : "ConstantScore(IntersectsPrefixTreeQuery(fieldName=shape,queryShape=Pt(x=-81.08111523359743,y=32.04772418111284),detailLevel=21,prefixGridScanLevel=20))^0.0",
          "details" : [ ]
        }
      },
      {
        "_shard" : "<my_index>[0]",
        "_node" : "lrSfQEyVTWmWU828O6Qdsw",
        "_index" : "<my_index>",
        "_type" : "_doc",
        "_id" : "dI2O9XcBlBVQyhnplhLN",
        "_score" : 0.0,
        "_source" : {
          "shape" : {
            "coordinates" : [
              [
                [
                  -81.0809327358636,
                  32.0478081852515
                ],
                [
                  -81.0810333624468,
                  32.0475470233845
                ],
                [
                  -81.0811253535251,
                  32.0475727349866
                ],
                [
                  -81.0810247260436,
                  32.0478338967803
                ],
                [
                  -81.0809327358636,
                  32.0478081852515
                ]
              ]
            ],
            "type" : "Polygon"
          }
        },
        "_explanation" : {
          "value" : 0.0,
          "description" : "ConstantScore(IntersectsPrefixTreeQuery(fieldName=shape,queryShape=Pt(x=-81.08111523359743,y=32.04772418111284),detailLevel=21,prefixGridScanLevel=20))^0.0",
          "details" : [ ]
        }
      },
      {
        "_shard" : "[<my_index>][1]",
        "_node" : "8jO4hXBuQL-cGobekTsjwg",
        "_index" : "<my_index>",
        "_type" : "_doc",
        "_id" : "cI2O9XcBlBVQyhnplhLN",
        "_score" : 0.0,
        "_source" : {
          "shape" : {
            "coordinates" : [
              [
                [
                  -81.0811167162237,
                  32.0478596090633
                ],
                [
                  -81.0812173428069,
                  32.0475984458201
                ],
                [
                  -81.0813093320886,
                  32.0476241574079
                ],
                [
                  -81.0812087064037,
                  32.0478853205776
                ],
                [
                  -81.0811167162237,
                  32.0478596090633
                ]
              ]
            ],
            "type" : "Polygon"
          }
        },
        "_explanation" : {
          "value" : 0.0,
          "description" : "ConstantScore(IntersectsPrefixTreeQuery(fieldName=shape,queryShape=Pt(x=-81.08111523359743,y=32.04772418111284),detailLevel=21,prefixGridScanLevel=20))^0.0",
          "details" : [ ]
        }
      }
    ]
  }
}
enter code here
Improve this question
2
  • Can you share better examples? The two polygons from above are definitely not intersecting: prnt.sc/11aqmq8 Commented Apr 12, 2021 at 16:44
  • 1
    We may have found the problem and are currently re-indexing. We toggled some advanced settings (coerce) on the geo_shape mapping and it looks like setting that also sets the strategy to "recursive" for whatever reason. I'll post it as the answer if we can confirm it is the problem. Commented Apr 12, 2021 at 18:45

1 Answer 1

Reset to default
0

It turns out that the mapping for our shape field was explicitly being set with the property strategy: "recursive".

When we created the component template mapping for the field, we set the 'ignore malformed' to true under the 'advanced settings' in Kibana. Whenever we loaded data into the index it automatically used the old tree structure. This must be a bug as you would not expect setting one of the advanced settings would set the tree type. I was able to replicate the behavior with a new mapping and index.

Since we wanted to keep the 'ignore malformed' option, I recreated the mapping by loading the json with:

"shape": {
   "type": "geo_shape",
   "ignore_malformed": true,
}

This preserved our option, and when we loaded data to the index, it was using the default tree. We were able to confirm this by running our previous searches which were now highly accurate (inches, if not more).

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.