Check null column value and replace it with a CONSTANT in Spark DataFrame

Question

I am new to Spark and Scala.

I have created a DataFrame from a csv in Spark. There is a column in the generated DataFrame which has a null value for some rows.

I want to check for this null value and replace it with a constant word say "Hello".

How can I do this in Spark?

Here is my sample code to create a dataframe from csv.

val DFCsv = spark.read.format("csv") .option("sep", ',') .option("inferSchema", "true") .option("header", "true") .load("/tmp/my.csv")
 DFCsv.show()

Now one of the columns in that dataframe named "id" is null or empty/blank for some of the rows.

How can I iterate over each row one by one and then fill out column named "id" with a constant "Hello" String.

Possible duplicate of Replace null values in Spark DataFrame — user10938362
– user10938362, Commented Jan 29, 2019 at 12:37

Chaitanya · Accepted Answer · 2019-01-29 12:38:24Z

-2

One of the easiest way is to surround the null value with a Option and then do a pattern match on it.

Option(null) gets converted to None
Option(null).getOrElse("Hello)

answered Jan 29, 2019 at 12:38

Chaitanya

3,65417 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user1393608 Over a year ago

Here is my sample code to create a dataframe from csv. val DFCsv = spark.read.format("csv") .option("sep", ',') .option("inferSchema", "true") .option("header", "true") .load("/tmp/my.csv") DFCsv.show() Now one of the columns in that dataframe named "id" is null or empty/blank for some of the rows.How can I iterate over each row one by one and then fill out column named "id" with a constant "Hello" String.

Collectives™ on Stack Overflow

Check null column value and replace it with a CONSTANT in Spark DataFrame

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related