How to extract data from Mapstruct in Spark DataFrame?

Question

How to extract data from df_raw in col("label") which is Mapstruct?

I'm using Spark 1.6. I got data from Hive by hivesql in Spark, then I got a dataframe, but one column in dataframe is Mapstruct, I tried to extract data from it but failed, hope some help from stackoverflow, 3Q very much.

After I got data from Hive, I got a dataframe named df_raw, the schema is :

root
 |-- subscriberid: string (nullable = true)
 |-- time: string (nullable = true)
 |-- itemid: string (nullable = true)
 |-- label: map (nullable = true)
 |    |-- key: string
 |    |-- value: string (valueContainsNull = true)
 |-- partitiondate: string (nullable = true)

and df_raw.show(3) is :

+------------+-------------------+------+--------------------+-------------+
|subscriberid|               time|itemid|               label|partitiondate|
+------------+-------------------+------+--------------------+-------------+
|     1569960|2019-09-08 08:00:01| 46611|Map(license -> yo...|     20190908|
|     1104555|2019-09-08 08:00:29| 46445|Map(license -> wa...|     20190908|
|     1309036|2019-09-08 08:00:55| 45219|Map(license -> yo...|     20190908|
+------------+-------------------+------+--------------------+-------------+

In order to get it clearly, I transform df_raw to rdd and take 2 data from it:

val rawRDD: RDD[String] = df_raw.rdd.map(pojo => pojo.mkString("\t"))
println("——————————" + "\n")
rawRDD.take(2).foreach(println)

the data is:

1545807 2019-09-10 07:29:41 4706    Map(license -> wa, videoid -> 4706, mediapaytype -> 1, duration -> 131) 20190908
1496840 2019-09-10 07:30:43 4535    Map(license -> you, videoid -> 4535, mediapaytype -> 1, duration -> 137)    20190908

I wanna know how to extract data from df_raw in col("label") separately?

I tried to get a new dataframe like this:

  val df_userBehaviorsRow_1 = rawUserBehaviorsData.map(line => {
    val splits = line.split("\t")

    val subscriberid = splits(0)
    val time= splits(1)
    val itemid = splits(2)

    val label = splits(3)
    val resultant = label.map{m=>
      val seq=m.values.toSeq
      (seq(0),seq(1),seq(2))
    }

    val license = resultant._1
    val duration = resultant._3

    (subscriberid , time, itemid, label, license，duration)
  }).toDF

I failed, and IntelliJ IDEA can't even recognize "val resultant = label.map{m=>val seq=m.values.toSeq(seq(0),seq(1),seq(2))}"

Hope some help please, 3Q very much.

firsni · Accepted Answer · 2019-09-12 07:38:18Z

1

In order to select for example the license values in a column, you just select the column and apply the key license.

import org.apache.spark.sql.functions.sql.col
df_raw.select(col("label")("license")).show()

you can use withColumn to add your column license to the dataframe

 df_raw_new = df_raw.withColumn("license", col("label")("license").alias("license"))

edited Sep 12, 2019 at 7:38

answered Sep 11, 2019 at 14:48

firsni

9266 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

konverse Over a year ago

It's very nice of U, 3Q. It works but with a little change. val df_raw_new = df_raw.withColumn("license", col("label")("license").alias("license")) Thank U very much

konverse Over a year ago

I'm so sorry, I did put+1, but the system showed "Thanks for the feedback! Votes cast by those with less than 15 reputation are recorded, but do not change the publicly displayed post score." (‧_‧?)

konverse Over a year ago

The system also showed me "End a line with two spaces to add a <br/> linebreak:" . I tried to mark my comment as good, but it didn't work, I'd try it later~ Thank U for your help~^o^

Collectives™ on Stack Overflow

How to extract data from Mapstruct in Spark DataFrame?

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related