Data manipulation on all columns in Dataset with Java API

Question

After reading csv file in Dataset, want to remove spaces from String type data using Java API.

Apache Spark 2.0.0

Dataset<Row> dataset = sparkSession.read().format("csv").option("header", "true").load("/pathToCsv/data.csv");
Dataset<String> dataset2 = dataset.map(new MapFunction<Row,String>() {

    @Override
    public String call(Row value) throws Exception {

        return value.getString(0).replace(" ", ""); 
        // But this will remove space from only first column
    }
}, Encoders.STRING());

By using MapFunction, not able to remove spaces from all columns.

But in Scala, by using following way in spark-shell able to perform desired operation.

val ds = spark.read.format("csv").option("header", "true").load("/pathToCsv/data.csv")
val opds = ds.select(ds.columns.map(c => regexp_replace(col(c), " ", "").alias(c)): _*)

Dataset opds have data without spaces. Want to achieve same in Java. But in Java API columns method returns String[] and not able to perform functional programming on Dataset.

Input Data

+----------------+----------+-----+---+---+
|               x|         y|    z|  a|  b|
+----------------+----------+-----+---+---+
|     Hello World|John Smith|There|  1|2.3|
|Welcome to world| Bob Alice|Where|  5|3.6|
+----------------+----------+-----+---+---+

Expected Output Data

+--------------+---------+-----+---+---+
|             x|        y|    z|  a|  b|
+--------------+---------+-----+---+---+
|    HelloWorld|JohnSmith|There|  1|2.3|
|Welcometoworld| BobAlice|Where|  5|3.6|
+--------------+---------+-----+---+---+

At which position you want to remove space, post a sample string and output you are expecting. You can use trim() function to remove leading and trailing white space. — Ravikumar
– Ravikumar, Commented Aug 4, 2016 at 12:41
You can use regex for removing spaces between Strings, Just post a sample string and what output string you are expecting after removing spaces. — Ravikumar
– Ravikumar, Commented Aug 4, 2016 at 13:15

user6022341 · Accepted Answer · 2016-08-04 13:34:09Z

3

Try:

for (String col: dataset.columns) {
  dataset = dataset.withColumn(col, regexp_replace(dataset.col(col), " ", ""));
}

answered Aug 4, 2016 at 13:34

community wiki

user6022341

Sign up to request clarification or add additional context in comments.

Comments

Community · Accepted Answer · 2017-05-23 12:25:45Z

0

You can try following regex to remove white spaces between strings.

value.getString(0).replaceAll("\\s+", "");

About \s+ : match any white space character between one and unlimited times, as many times as possible. Instead of replace use replaceAll function.

More about replace and replaceAll functions Difference between String replace() and replaceAll()

edited May 23, 2017 at 12:25

CommunityBot

11 silver badge

answered Aug 4, 2016 at 13:34

Ravikumar

90112 silver badges22 bronze badges

3 Comments

Sheel Over a year ago

this will remove space from first column only.

Ravikumar Over a year ago

Print out value.getString(0) and post it so that to see whether string is multiline.

Ravikumar Over a year ago

@mastersheel007 try this value.getString(0).replaceAll("(?is)\\s+", "");

Collectives™ on Stack Overflow

Data manipulation on all columns in Dataset with Java API

2 Answers 2

Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related