pyspark dataframe filtering on multiple columns

Question

I have a pyspark dataframe which looks like below

df

num11   num21 
10     10
20     30 
5      25

I am filtering above dataframe on all columns present, and selecting rows with number greater than 10 [no of columns can be more than two]

from pyspark.sql.functions import col
col_list = df.schema.names
df_fltered = df.where(col(c) >= 10 for c in col_list)

desired output is :

num11    num21
10       10
20       30

How can we achieve filtering on multiple columns using iteration on column list as above. [all efforts are appriciated]

[error i reveive is : condition should be string or column]

ags29 · Accepted Answer · 2017-11-17 17:47:44Z

2

As an alternative, if you not averse to some sql-like snippets of code, the following should work:

df.where("AND".join(["(%s >=10)"%(col) for col in col_list]))

answered Nov 17, 2017 at 17:47

ags29

2,7061 gold badge11 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

akuiper · Accepted Answer · 2017-11-16 21:15:12Z

1

You can use functools.reduce to combine the column conditions, to simulate an all condition, for instance, you can use reduce(lambda x, y: x & y, ...):

import pyspark.sql.functions as F
from functools import reduce

df.where(reduce(lambda x, y: x & y,  (F.col(x) >= 10 for x in df.columns))).show()
+-----+-----+
|num11|num21|
+-----+-----+
|   10|   10|
|   20|   30|
+-----+-----+

answered Nov 16, 2017 at 21:15

akuiper

216k33 gold badges362 silver badges379 bronze badges

2 Comments

anurag Over a year ago

Thanks for the reply. I tried above logic as df_pass = df3.where(reduce(lambda x, y: x | y, (col(x).cast('integer') == 2 | col('status1').cast('string') == 'present in source only' for x in columns1))) but i am receiving error when i am passing 2 columns Error received is "Py4JException: Method or([class java.lang.Integer]) does not exist"

akuiper Over a year ago

You need to wrap the comparison in parenthesis.

df3.where(reduce(lambda x, y: x | y, ((col(x).cast('integer') == 2) | (col('status1').cast('string') == 'present in source only') for x in columns1)))

because of the higher operator precedence of | compared to ==.

Collectives™ on Stack Overflow

pyspark dataframe filtering on multiple columns

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related