How to pass list in Pyspark function "Withcolumn"

Question

I am doing ltrim and rtrim on multiple columns of dataframe but now i am able to do it individually . like

# selected_colums = selected_colums.withColumn("last_name", ltrim(selected_colums.last_name))
# selected_colums = selected_colums.withColumn("last_name", rtrim(selected_colums.last_name))

# selected_colums = selected_colums.withColumn("email", ltrim(selected_colums.email))
# selected_colums = selected_colums.withColumn("email", rtrim(selected_colums.email))

# selected_colums = selected_colums.withColumn("phone_number", ltrim(selected_colums.phone_number))
# selected_colums = selected_colums.withColumn("phone_number", rtrim(selected_colums.phone_number))

But I want to do it in loop like below

sdk = ['first_name','last_name','email','phone_number','email_alt','phone_number_alt']
for x in sdk:
  selected_colums = selected_colums.withColumn(x, ltrim(selected_colums.last_name))

Its giving me syntax error. Please help me to optimize this code so that for any number of column i can able to do ltrim or rtrim just passing list.

s.polam · Accepted Answer · 2020-11-15 14:14:41Z

1

Check below code.

Import required functions

>>> from pyspark.sql.functions import col

Apply ltrim and rtrim on all columns

>>> columnExprs = map(lambda c: rtrim(ltrim(col(c))).alias(c),df.columns)

Apply columnExprs in select

df.select(*columnExprs).show()

answered Nov 15, 2020 at 14:14

s.polam

10.4k2 gold badges17 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to pass list in Pyspark function "Withcolumn"

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related