Data transformation in Spark Scala

Question

I have the following dataframe

+-----+-----+-----+ .......+-------+
|item1|item2|item3|........| itemN |
+-----+-----+-----+........|-------+
|   v1|   v2|   v3|........| vN----+
|   v4|   v5|   v6|........| v2N---+
+-----+-----+-----+........|-------+

here item1 , item2 and item3 are the column names and table consists of 1 row v1,v2,v3.

I want to transform it into

colA   colB
item1    v1
item2    v2
item3    v3
 .        .
 .        .
 .        .

Here there are two columns lets say colA and colB and rows are as shown.

How to do this transformation in spark using scala?

stackoverflow.com/questions/35603689/…. It is similar to the answer provided as java code. Can you help me with that in scala as I am new to scala? — satyambansal117
– satyambansal117, Commented Oct 18, 2016 at 8:30

Tzach Zohar · Accepted Answer · 2016-10-18 09:27:33Z

5

You can use explode:

import org.apache.spark.sql.functions._

input.show()
// +-----+-----+-----+
// |item1|item2|item3|
// +-----+-----+-----+
// |   v1|   v2|   v3|
// |   v4|   v5|   v6|
// +-----+-----+-----+

val columns: Array[String] = input.columns

val result = input.explode(columns.map(s => col(s)): _*) {
  r: Row => columns.zipWithIndex.map { case (name, index) => (name, r.getAs[String](index)) }
}.select($"_1" as "colA", $"_2" as "colB")

result.show()
// +-----+----+
// | colA|colB|
// +-----+----+
// |item1|  v1|
// |item2|  v2|
// |item3|  v3|
// |item1|  v4|
// |item2|  v5|
// |item3|  v6|
// +-----+----+

edited Oct 18, 2016 at 9:27

answered Oct 18, 2016 at 8:42

Tzach Zohar

37.9k3 gold badges83 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

satyambansal117 Over a year ago

Thanks for the above code . I have one more question , If the number of columns(i.e. item1,item2 ......item n) in initial table are very large then how to do the above thing?

Tzach Zohar Over a year ago

How large? And why wouldn't the above code work for any (valid) number of columns?

satyambansal117 Over a year ago

I don't know the number of columns initially lets say there are 100 columns. I think the above code takes as case the variables i1,i2,i3 corresponding to columns so if there are many columns how will this work?

satyambansal117 Over a year ago

Thanks , If values v1,v2 .... are of complex types(say array) rather than string then what change I have to make in above code

Tzach Zohar Over a year ago

Are they all the same type at least? If they are, just change r.getAs[String] to r.getAs[T] where T is the type you expect, for example it could be Array[String] or Array[(Int, String)] or whatever...

|

Collectives™ on Stack Overflow

Data transformation in Spark Scala

1 Answer 1

7 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related