Spark combine columns as nested array

Question

How can I combine columns in spark as a nested array?

val inputSmall = Seq(
    ("A", 0.3, "B", 0.25),
    ("A", 0.3, "g", 0.4),
    ("d", 0.0, "f", 0.1),
    ("d", 0.0, "d", 0.7),
    ("A", 0.3, "d", 0.7),
    ("d", 0.0, "g", 0.4),
    ("c", 0.2, "B", 0.25)).toDF("column1", "transformedCol1", "column2", "transformedCol2")

To something similar as

+-------+---------------+---------------+------- +
|column1|transformedCol1|transformedCol2|combined|
+-------+---------------+---------------+------ -+
|      A|            0.3|            0.3[0.3, 0.3]|
+-------+---------------+---------------+-------+

Daniel de Paula · Accepted Answer · 2016-12-20 10:26:02Z

29

If you want to combine multiple columns into a new column of ArrayType, you can use the array function:

import org.apache.spark.sql.functions._
val result = inputSmall.withColumn("combined", array($"transformedCol1", $"transformedCol2"))
result.show()

+-------+---------------+-------+---------------+-----------+
|column1|transformedCol1|column2|transformedCol2|   combined|
+-------+---------------+-------+---------------+-----------+
|      A|            0.3|      B|           0.25|[0.3, 0.25]|
|      A|            0.3|      g|            0.4| [0.3, 0.4]|
|      d|            0.0|      f|            0.1| [0.0, 0.1]|
|      d|            0.0|      d|            0.7| [0.0, 0.7]|
|      A|            0.3|      d|            0.7| [0.3, 0.7]|
|      d|            0.0|      g|            0.4| [0.0, 0.4]|
|      c|            0.2|      B|           0.25|[0.2, 0.25]|
+-------+---------------+-------+---------------+-----------+

edited Dec 20, 2016 at 10:26

answered Dec 20, 2016 at 10:19

Daniel de Paula

18k9 gold badges75 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Max Murphy Over a year ago

What if I have an array of columns to be combined? val names=Seq("foo", "bar") followed by withColumn("combined", array(names:_*)) is unsupported. That means that if the names change dynamically there seems to be no way of achieving this.

Max Murphy Over a year ago

Aha, it will accept multiple columns, as opposed to multiple strings, so this works: val names = Seq("foo", "bar"); frame.withColumn("combined", array(names.map(frame(_)):_*))

NNM · Accepted Answer · 2021-10-29 19:15:40Z

0

I would rather use the below, will handle column changes

inputSmall.withColumn("combined", F.array("*"))

answered Oct 29, 2021 at 19:15

NNM

4282 silver badges12 bronze badges

1 Comment

Ani Menon Over a year ago

This doesn't work. Throws pyspark.sql.utils.AnalysisException: "cannot resolve 'array( ...

Collectives™ on Stack Overflow

Spark combine columns as nested array

2 Answers 2

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related