pyspark dataframe make the map values using column name

Question

I have data frame as below first one in pyspark dataframe.
Without change to pandas Df, I need to chage the dataframe like the second one with map values.
Does anyone have an idea for this job?

    a   b
A   1   2
B   4   2

↓
    value
A   {a : 1, b : 2}
B   {a : 4, b: 2}

I know, I'm suggesting you find the equivalent. Even knowing the pandas equivalent should help you. — smci
– smci, Commented Nov 9, 2021 at 4:50

Nithish · Accepted Answer · 2021-11-09 09:26:38Z

1

You can create a struct and then convert it into a JSON field using to_json method.

Working Example

import pyspark.sql.functions as F

df = spark.createDataFrame([{"a": 1, "b": 2}, {"a": 4, "b": 2}])

df.withColumn("value", F.to_json(F.struct(*[F.col(c).alias(c) for c in df.columns]))).select("value").show()

Output

+-------------+
|        value|
+-------------+
|{"a":1,"b":2}|
|{"a":4,"b":2}|
+-------------+

answered Nov 9, 2021 at 9:26

Nithish

3,2572 gold badges11 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

pyspark dataframe make the map values using column name

1 Answer 1

Working Example

Output

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Working Example

Output

Comments

Your Answer

Sign up or log in

Post as a guest

Related