Filter Scala dataframe by column of arrays

Question

My scala dataframe has a column that has the datatype array(element: String). I want to display those rows of the dataframe that has the word "hello" in that column.

I have this:

display(df.filter($"my_column".contains("hello")))

I get an error because of data mismatch. It says that argument 1 requires string type, however, 'my:column' is of array<string> type.

try array_contains function

undefined_variable
– undefined_variable

2018-09-07 11:53:46 +00:00
Commented Sep 7, 2018 at 11:53 — undefined_variable
– undefined_variable, Commented Sep 7, 2018 at 11:53

Manoj Kumar Dhakad · Accepted Answer · 2018-09-07 20:30:29Z

1

You can use array_contains function

import org.apache.spark.sql.functions._

df.filter(array_contains(df.col("my_column"), "hello")).show

answered Sep 7, 2018 at 20:30

Manoj Kumar Dhakad

1,9021 gold badge14 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Filter Scala dataframe by column of arrays

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related