Spark filter on dataframe with array containing a map

Question

I have a dataframe with schema which has a nested array of map values:

root
 |-- array_of_properties: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- name: string (nullable = true)
 |    |    |-- props: map (nullable = true)
 |    |    |    |-- key: string
 |    |    |    |-- value: string (valueContainsNull = true)

I need to filter on the struct name and some specific key's values in the map inside the array. I can filter on the name:

dataframe.filter(array_contains(col("array_of_properties.name"), "somename"))

How do I add AND filters on values of two keys in the nested props map (for example a key name is_enabled with a boolean value of true or false, and a key name of source with a string value of test) ?

blackbishop · Accepted Answer · 2021-12-21 17:39:30Z

3

Use exists function:

dataframe.filter("exists(array_of_properties, x -> x.name = 'somename' and x.props['is_enabled'] is true)")

answered Dec 21, 2021 at 17:39

blackbishop

32.8k11 gold badges61 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

skisurfsup Over a year ago

That works - thank you. Are there any drawbacks to using an exists filter ?

Collectives™ on Stack Overflow

Spark filter on dataframe with array containing a map

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related