Pyspark: Fill a fix value in pyspark column if other column is null

Question

I have a pyspark dataframe which has two columns. I want to populate one column with a fix value if row value in other column is null. So in customer_df if customer_address is null then populate city column as 'unknown'

I am trying this

customer_df = customer_df.withColumn('city',when(customer_df.customer_address == '','unknown')

But this gives syntax error. What is that I am missing here? Thanks in advance

Daeho Ro · Accepted Answer · 2020-09-26 02:21:26Z

3

customer_df = customer_df.withColumn('city', 
    when(col(customer_address).isNull(), 'unknown').otherwise(col('city'))
  )

answered Sep 26, 2020 at 2:21

Daeho Ro

13.7k4 gold badges25 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pyspark: Fill a fix value in pyspark column if other column is null

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related