Spark Java edit data in column

Question

I would like to iterate through the content of a column in a spark DataFrame and correct the data within a cell if it meets a certain condition

+-------------+
|column_title |
+-------------+
+-----+
|null |
+-----+
+-----+
|0    |
+-----+
+-----+
|1    |
+-----+

Lets say I want to display something else when value of column is null, I tried with

Column.when() DataSet.withColumn()

But I cant find the right method, i don't think it would be necessary to convert to RDD and iterate through it.

abaghel · Accepted Answer · 2019-02-13 15:06:49Z

4

You can use when and equalTo or when and isNull.

Dataset<Row> df1 = df.withColumn("value", when(col("value").equalTo("bbb"), "ccc").otherwise(col("value")));

Dataset<Row> df2 = df.withColumn("value", when(col("value").isNull(), "ccc").otherwise(col("value")));

If you only want to replace null values then you can also use na and fill.

Dataset<Row> df3 = df.na().fill("ccc");

edited Feb 13, 2019 at 15:06

answered Feb 13, 2019 at 14:56

abaghel

15.4k2 gold badges56 silver badges69 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

SPS Over a year ago

Can we do df.withColumn("*",..) if we intend to run the replacement in all the columns?

Nimantha · Accepted Answer · 2021-12-06 07:47:41Z

1

Another way of doing this could be by using UDF.

Create a UDF:

private static UDF1 myUdf = new UDF1<String, String>() {
public String call(final String str) throws Exception {
    // any condition or custom function can be used
    return StringUtils.rightPad(str, 25, 'A');
  }
};

Register UDF in SparkSession:

sparkSession.udf().register("myUdf", myUdf, DataTypes.StringType);

Apply udf on dataset:

Dataset<Row> dataset = dataset.withColumn("city", functions.callUDF("myudf", col("city")));

edited Dec 6, 2021 at 7:47

Nimantha

6,5376 gold badges32 silver badges78 bronze badges

answered Oct 14, 2019 at 16:26

Aryan087

5266 silver badges26 bronze badges

Collectives™ on Stack Overflow

Spark Java edit data in column

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related