Transforming a dataframe in spark scala

Question

I have a dataframe (df) of the following form:

+-----+-----   +
|id   |items   |
+-----+-----   +
|   0 |  item1 |
|   1 |  item2 |
+-----+-----   +

Here first column id is an int and second column items is of type struct. Lets say item is as shown:

     item1
        |-a
        |-b
        |-c
        |-d

I want the resultant table of the form

   +-----+-----   +
   |id   |col2   |
   +-----+-----   +
   |   0 |  a    |
   |   0 |  b    |
   |   0 |  c    |
   |   0 |  d    |
   |   1 |  a    |
   |   1 |  b    |
   |   1 |  c    |
   |   1 |  d    |
   +-----+-----   +

I want to expand struct for every column?
How to do it?

Shivansh · Accepted Answer · 2016-10-20 19:02:30Z

1

This peice of code may solve your problem:

df.rdd.flatMap{row=>
val id=row.getInt(0)
val arrayOfString=row.getAs[Array[String]](1)
arrayOfString.map(value=>(id,value)
}.toDF("id","col2")

Note: this code is not tested !

answered Oct 20, 2016 at 19:02

Shivansh

3,56426 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

satyambansal117 Over a year ago

In this the type of column "items" is struct . How can it be type casted to array[string]

Shivansh Over a year ago

Then remove the array[String] part and put your own value in which you want to take out

Collectives™ on Stack Overflow

Transforming a dataframe in spark scala

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related