Check for list of substrings inside string column in PySpark

Question

For checking if a single string is contained in rows of one column. (for example, "abc" is contained in "abcdef"), the following code is useful:

df_filtered = df.filter(df.columnName.contains('abc'))

The result would be for example "_wordabc","thisabce","2abc1".

How can I check for multiple strings (for example ['ab1','cd2','ef3']) at the same time?

I'm ideally searching for something like this:

df_filtered = df.filter(df.columnName.contains(['word1','word2','word3']))

The result would be for example "x_ab1","_cd2_","abef3".

Please, post scalable solutions (no for loops, for example) because the aim is to check a big list around 1000 elements.

User12345 · Accepted Answer · 2021-10-07 17:52:34Z

2

All you need is isin

df_filtered = df.filter(df['columnName'].isin('word1','word2','word3')

Edit

You need rlike function to achieve your result

words="(aaa|bbb|ccc)"

df.filter(df['columnName'].rlike(words))

edited Oct 7, 2021 at 17:52

answered Oct 7, 2021 at 17:10

User12345

5,54015 gold badges62 silver badges112 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Manrique Over a year ago

Thanks, but this isn't what I'm aiming for. I'll clarify my question because i meant "contains" not "equal".

User12345 Over a year ago

@Manrique Please check edit part now

Collectives™ on Stack Overflow

Check for list of substrings inside string column in PySpark

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related