Spark Java - Collect multiple columns into array column

Question

I have a dataframe with multiple columns:

| a | b | c | d |
-----------------
| 0 | 4 | 3 | 6 |
| 1 | 7 | 0 | 4 |
| 2 | 4 | 3 | 6 |
| 3 | 9 | 5 | 9 |

I would now like to combine [b,c,d] into a single column. However, I do not know, how big the list of columns will be, otherwise I could just use a UDF3 to combine the three.

So the desired outcome is:

| a | combined  |
-----------------
| 0 | [4, 3, 6] |
| 1 | [7, 0, 4] |
| 2 | [4, 3, 6] |
| 3 | [9, 5, 9] |

How can I achieve this?

Non-working pseudo-code:

public static Dataset<Row> mergeColumns(Dataset<Row> ds, List<String> columns) {
   return ds.withColumn("combined", collectAsList(columns))
}

Worst-case workaround would be a switch statement on the number of input columns and then write a UDF each for, i.e. 2-20 input columns and throw an error, if more input columns are supplied.

use array inbuilt function

Anahcolus
– Anahcolus

2018-07-05 07:09:18 +00:00
Commented Jul 5, 2018 at 7:09 — Anahcolus
– Anahcolus, Commented Jul 5, 2018 at 7:09

Grisha Weintraub · Accepted Answer · 2018-07-05 07:34:57Z

5

As Ramesh mentioned in his comment, you can use array function. You only need to convert your columns list to Column array.

public static Dataset<Row> mergeColumns(Dataset<Row> ds, List<String> columns) {
    return ds.withColumn("combined", functions.array(columns.stream().map(functions::col).toArray(Column[]::new)))
}

answered Jul 5, 2018 at 7:34

Grisha Weintraub

7,9961 gold badge28 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Chandan Ray · Accepted Answer · 2018-07-05 07:36:07Z

0

val newDF = df.drop("a")

//create a new dataframe to get the combination of columns except the first column
val df2 = df.withColumn("combined", concat_ws(",",newDF.columns.map(collist => 
col(collist)): _*))

//new dataframe to select only the first and combined columns
val columnNames = Seq("a", "combined")
df2.select(columnNames.head, columnNames.tail:_*).show()

answered Jul 5, 2018 at 7:36

Chandan Ray

2,0911 gold badge13 silver badges16 bronze badges

1 Comment

Carl Ambroselli Over a year ago

That would result in a column of StringType, rather than an array, or?

Collectives™ on Stack Overflow

Spark Java - Collect multiple columns into array column

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related