How to extract key/value from maps into separate columns in spark sql

Using the same input from other answer

val df = Seq(
    (Map("timestamp1"->1585008000, "timestamp3"-> 1584921600), "AFN"),
    (Map("timestamp2"-> 1584835200), "AFN"),
    (null, "AFN") 
).toDF("addedSkuWithTimestamp", "skuType")

Try using explode, I tested this in spark 2.2.1 and 2.3.1

df.select(explode($"addedSkuWithTimestamp")).show(false)
+----------+----------+
|key       |value     |
+----------+----------+
|timestamp1|1585008000|
|timestamp3|1584921600|
|timestamp2|1584835200|
+----------+----------+

First let's create some data:

val df = Seq(
    (Map("sku1"->"timestamp1"), "AFN"),
    (Map("sku2"->"timestamp2"), "AFN"),
    (null, "AFN") 
).toDF("addedSkuWithTimestamp", "skuType")
.show(false)

+---------------------+-------+
|addedSkuWithTimestamp|skuType|
+---------------------+-------+
| [sku1 -> timestamp1]|    AFN|
| [sku2 -> timestamp2]|    AFN|
|                 null|    AFN|
+---------------------+-------+

This will have the following schema:

scala> df.printSchema()

root
 |-- addedSkuWithTimestamp: map (nullable = true)
 |    |-- key: string
 |    |-- value: string (valueContainsNull = true)
 |-- skuType: string (nullable = true)

Spark < 2.3

The next code will extract the columns sku_key and sku_value from addedSkuWithTimestamp column using the mapToTupleUDF udf function:

val mapToTupleUDF = udf((sku: Map[String, String]) => if(sku != null) sku.toSeq(0) else null)

df.withColumn("addedSkuWithTimestamp", mapToTupleUDF($"addedSkuWithTimestamp"))
  .withColumn("Sku", when($"addedSkuWithTimestamp".isNotNull, $"addedSkuWithTimestamp._1"))
  .withColumn("Timestamp", when($"addedSkuWithTimestamp".isNotNull, $"addedSkuWithTimestamp._2"))
  .show(false)

+---------------------+-------+----+----------+
|addedSkuWithTimestamp|skuType|Sku |Timestamp |
+---------------------+-------+----+----------+
|[sku1, timestamp1]   |AFN    |sku1|timestamp1|
|[sku2, timestamp2]   |AFN    |sku2|timestamp2|
|null                 |AFN    |null|null      |
+---------------------+-------+----+----------+

Note that we can access addedSkuWithTimestamp._1 only if addedSkuWithTimestamp is not null.

Spark >= 2.3

Since Spark 2.3.0 you can use the build-in map_values and map_keys:

df.withColumn("Sku", map_keys($"addedSkuWithTimestamp").getItem(0))
  .withColumn("Timestamp", map_values($"addedSkuWithTimestamp").getItem(0))
  .show(false)

Output:

+---------------------+-------+----+----------+
|addedSkuWithTimestamp|skuType|Sku |Timestamp |
+---------------------+-------+----+----------+
|[sku1 -> timestamp1] |AFN    |sku1|timestamp1|
|[sku2 -> timestamp2] |AFN    |sku2|timestamp2|
|null                 |AFN    |null|null      |
+---------------------+-------+----+----------+