transform a column of array of string in pyspark to multiple rows in one column

Question

I would like to transform a table with a column as array of strings on databricks pyspark.

my table:

 id         values  (array<string>)
 rgf        ['vwervfrev', 'fweccf', 'tuyhert']
 rty        ['evvverws', 'ilonmunt', 'cedcrhb']

What I need:

 id         values
 rdf        'vwervfrev'
 rdf        'fweccf'
 rdf        'tuyhert'
 rty        'evvverws'
 rty        'ilonmunt'
 rty        'cedcrhb'

I am not sure how to do the transformation ?

thanks

spark.apache.org/docs/latest/api/python/…

akuiper
– akuiper

2019-12-16 23:39:13 +00:00
Commented Dec 16, 2019 at 23:39 — akuiper
– akuiper, Commented Dec 16, 2019 at 23:39

Shadowtrooper · Accepted Answer · 2019-12-17 08:59:39Z

1

You can do this with the explode function:

from pyspark.sql.functions import explode, col

new_df = df.withColumn("values", explode(col("values")))
new_df.show()

https://spark.apache.org/docs/latest/api/python/_modules/pyspark/sql/functions.html#explode

answered Dec 17, 2019 at 8:59

Shadowtrooper

1,47216 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

transform a column of array of string in pyspark to multiple rows in one column

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related