Replacing null values in a column in Pyspark Dataframe

Question

I need to replace null values present in a column in Spark dataframe. Below is the code I tried

df=df.na.fill(0,Seq('c_amount')).show()

But it is throwing me an error NameError: name 'Seq' is not defined

Below is my table

   +------------+--------+
   |c_account_id|c_amount|
   +------------+--------+ 
   |           1|    null|    
   |           2|    123 |
   |           3|    null|
   +------------+--------+

Expected output

   +------------+--------+
   |c_account_id|c_amount|
   +------------+--------+ 
   |           1|       0|    
   |           2|     123|
   |           3|       0|
   +------------+--------+

dsk · Accepted Answer · 2020-07-02 13:48:18Z

1

You need to use like this

df = df.fillna("<BLANK>", subset=['col_name'])

answered Jul 2, 2020 at 13:48

dsk

2,0272 gold badges12 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Tommy_SK Over a year ago

I am getting <BLANK> in the null place.Should I use "0" there ? If I use that won't that be considered as string ?

dsk Over a year ago

yes please.use 0. can you also please mind approving the answer ..

Tommy_SK Over a year ago

Sir After applying that code I am able to get correct answer.But after that if I try printSchema() it says 'NoneType' object has no attribute 'printSchema'. Can you elaborate on the problem Sir ?

dsk Over a year ago

NoneType means that instead of an instance of whatever Class or Object you think you're working with, you've actually got None. That usually means that an assignment or function call up above failed or returned an unexpected result.

Tommy_SK Over a year ago

But why I get such error after using this function ? Is there a way to solve this ?

|

Collectives™ on Stack Overflow

Replacing null values in a column in Pyspark Dataframe

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related