scala - how to concatenate columns of a DataFrame with concat_ws?

Question

I want to concatenate columns of my Dataframe. I wrote a Udf to achieve that but as I can see concat_ws expects columns while I'm passing it Array[String]. How should I pass columns of my DataFrame correctly?

val columns = salesDF.columns
val concatColumns = udf((arr: Seq[String]) => arr.mkString(" "))

salesDF.select(concat_ws(",", concatColumns(columns)))

eliasah · Accepted Answer · 2018-08-30 15:08:31Z

3

You will just need to pass the selection as an Array[Column]:

import org.apache.spark.sql.functions.{col,concat_ws}

val selection = salesDF.columns.map(col)
salesDF.select(concat_ws(",", selection : _*))

edited Aug 30, 2018 at 15:08

answered Aug 3, 2018 at 15:33

eliasah

40.5k12 gold badges128 silver badges159 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

scala - how to concatenate columns of a DataFrame with concat_ws?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related