Split the map into two columns pyspark

Question

I have a dataframe with a map:

sdf = spark.createDataFrame(
    [
        (1,  {'Kira':25,'Lilly':15}),  
        (2, {'Tom':14}),
    ],
    ["id", "label"]  
)
+---+-------------------------+
|id |label                    |
+---+-------------------------+
|1  |{Lilly -> 15, Kira -> 25}|
|2  |{Tom -> 14}              |
+---+-------------------------+

And I want to put the keys in one column and the values in another, like this:

+---+-----+---+
|id |name |age|
+---+-----+---+
|1  |Kira |25 |
|1  |Lilly|15 |
|2  |Tom  |14 |
+---+-----+---+

wwnde · Accepted Answer · 2022-06-14 11:14:27Z

1

Long hand. Use map collection functions to create name and age colunms. Leverage inline function to explode

sdf.withColumn('name',map_keys('label')).withColumn('age', map_values('label')).selectExpr('id','inline(arrays_zip(name,age))').show()

+---+-----+---+
| id| name|age|
+---+-----+---+
|  1|Lilly| 15|
|  1| Kira| 25|
|  2|  Tom| 14|
+---+-----+---+

answered Jun 14, 2022 at 11:14

wwnde

26.7k6 gold badges22 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Split the map into two columns pyspark

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related