Convert a scala dataframe Json string column to a map

Question

I have a dataframe structure as shown below:

+----+----+--------------------+
|col1|col2|col3                |
+----+----+--------------------+
|1   |2   |{"key1" : "val1"}   |
|3   |4   |{"key2" : "val2"}   |
+----+----+--------------------+

col3 is a Json String. I would like to add another column to the dataframe that will convert col3 to a MAP. Something like this:

+----+----+--------------------+----------------+
|col1|col2|col3                |col4            |
+----+----+--------------------+----------------+
|1   |2   |{"key1" : "val1"}   |[key1 -> val1]  |
|3   |4   |{"key2" : "val2"}   |[key2 -> val2]  |
+----+----+--------------------+----------------+

My Scala code snippet is something like this:

val df: DataFrame = getRowDataset(payload, schema, "payload") // df is represented by the first column
df.withColumn("col4", "<>") // <What do I need to do here?>

How am I supposed to implement this with Scala?

Please check this stackoverflow.com/questions/57044746/… related question — abiratsis
– abiratsis, Commented Jul 19, 2023 at 11:04
Does this answer your question? How to convert nested JSON to map object in scala — Mohana B C
– Mohana B C, Commented Jul 19, 2023 at 15:26
val df2 = df.withColumn("map_data", from_json((df("crawl_request")), mappingSchema)). This works as expected. — mang4521
– mang4521, Commented Jul 19, 2023 at 17:21

mang4521 · Accepted Answer · 2023-07-19 17:33:55Z

0

Keeping link as a reference, the solution is as follows:

    val mappingSchema = MapType(StringType, StringType)
    val df2 = df.withColumn("col4", from_json((df("col3")), mappingSchema))

Note: df("col3") is a Json String in my use case.

answered Jul 19, 2023 at 17:33

mang4521

8321 gold badge10 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Convert a scala dataframe Json string column to a map

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related