How to flatten a pyspark dataframe? (spark 1.6)

Question

I'm working with Spark 1.6

Here are my data :

eDF = sqlsc.createDataFrame([Row(v=1, eng_1=10,eng_2=20),
                        Row(v=2, eng_1=15,eng_2=30),
                        Row(v=3, eng_1=8,eng_2=12)])
eDF.select('v','eng_1','eng_2').show()

+---+-----+-----+
|  v|eng_1|eng_2|
+---+-----+-----+
|  1|   10|   20|
|  2|   15|   30|
|  3|    8|   12|
+---+-----+-----+

I would like to 'flatten' this table. That is to say :

+---+-----+---+
|  v|  key|val|
+---+-----+---+
|  1|eng_1| 10|
|  1|eng_2| 20|
|  2|eng_1| 15|
|  2|eng_2| 30|
|  3|eng_1|  8|
|  3|eng_2| 12|
+---+-----+---+

Note that since I'm working with Spark 1.6, I can't use pyspar.sql.functions.create_map or pyspark.sql.functions.posexplode.

akuiper · Accepted Answer · 2017-10-10 14:50:01Z

2

Use rdd.flatMap to flatten it:

df = spark.createDataFrame(
    eDF.rdd.flatMap(
        lambda r: [Row(v=r.v, key=col, val=r[col]) for col in ['eng_1', 'eng_2']]
    )
)
df.show()
+-----+---+---+
|  key|  v|val|
+-----+---+---+
|eng_1|  1| 10|
|eng_2|  1| 20|    
|eng_1|  2| 15|
|eng_2|  2| 30|
|eng_1|  3|  8|
|eng_2|  3| 12|
+-----+---+---+

answered Oct 10, 2017 at 14:50

akuiper

216k33 gold badges362 silver badges379 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to flatten a pyspark dataframe? (spark 1.6)

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related