Pyspark concat column to an array and explode values

Question

I am trying to transpose column to rows . First to concat columns into an array Second step is to explode the array column

Explode function is not working ..

>>> filteredPaths1.select(   array ( concat( col("v1.id"),lit(","),col("v2.id"),lit(",") ,col("v2.id") )).alias("test")     ).printSchema()
root
 |-- test: array (nullable = false)
 |    |-- element: string (containsNull = true)

Values in array column -

>>> filteredPaths1.select(   array ( concat( col("v1.id"),lit(","),col("v2.id"),lit(",") ,col("v2.id") )).alias("test")     ).show(10,False)
+--------------------------------------------------------------+                ]
|test                                                          |
+--------------------------------------------------------------+
|[Sorter_SAMPLE_CUSTOMER,Join_Source_Target,Join_Source_Target]|
+--------------------------------------------------------------+

However when trying to explode the array column it's not creating new rows , just giving the same output -

>>> filteredPaths1.select(   explode (array ( concat( col("v1.id"),lit(","),col("v2.id"),lit(",") ,col("v2.id") )).alias("test") )    ).show(10,False)
+------------------------------------------------------------+                  ]
|col                                                         |
+------------------------------------------------------------+
|Sorter_SAMPLE_CUSTOMER,Join_Source_Target,Join_Source_Target|
+------------------------------------------------------------+

Any reason explode is not working ?

concat creates a single string column out of your three.

cronoik
– cronoik

2020-06-20 16:05:57 +00:00
Commented Jun 20, 2020 at 16:05 — cronoik
– cronoik, Commented Jun 20, 2020 at 16:05

s.polam · Accepted Answer · 2020-06-20 15:45:00Z

1

Because of you are using array(concat(..)) means array of one value & exploding this you will get only one row, i.e same value.

Use split instead of array

filteredPaths1.select(explode(split(concat_ws(",",col("v1.id"),col("v2.id"),col("v2.id")),",")).alias("test"))

answered Jun 20, 2020 at 15:45

s.polam

10.4k2 gold badges17 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Pratik Rudra Over a year ago

Thanks for pointing it out , I have removed concat all together and it worked as well but thanks for providing solution .

Collectives™ on Stack Overflow

Pyspark concat column to an array and explode values

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related