Spark-Java: Concatenate aggregated groupBy's result

Question

This is what I am trying to do in Java but is not working as concat_ws requires Seq as the type of 2nd argument while collect_list return Column:

Dataset aggregatedDF = someDF
  .groupBy("colA")
  .agg(
      concat_ws(",", collect_list(col("colB"))).as("comma_sep_col_b")
  );

Same thing works in Scala:

val aggregatedDF = someDF
  .groupBy("colA")
  .agg(
      concat_ws(",", collect_list($"colB")) as "comma_sep_col_b"
  )

Works fine for me in java : Dataset aggregatedDF = df.groupBy("Age").agg(concat_ws(",", collect_list(col("Wage"))).as("comma_sep_col_b")); aggregatedDF.show(); — tarun
– tarun, Commented Nov 18, 2019 at 8:45

Sim · Accepted Answer · 2019-11-19 15:54:23Z

1

You probably want to use the array_join() function.

Background:

The signature of concat_ws in Scala includes variable arguments

def concat_ws(sep: String, exprs: Column*): Column

Variable arguments of type Column in Scala are handled as Seq[Column], with the Scala compiler providing the syntactic sugar management.

Note that the columns to be concatenated must be provided at transformation time, not execution time. collect_list() returns a single column, which is an ArrayType(...).

answered Nov 19, 2019 at 15:54

Sim

13.6k11 gold badges69 silver badges96 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spark-Java: Concatenate aggregated groupBy's result

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related