how to access map values and keys stored in a data frame in scala spark

Question

i have a table which description is as follows:

# col_name              data_type               comment             

id                      string                                      
persona_model           map<string,struct<score:double,tag:string>>                     

# Partition Information      
# col_name              data_type               comment             

process_date            string

sample row would be something like this(tab separated):

000000E91010441BB122402A45D439E7        {"Tech":{"score":0.21678,"tag":"OTHERS"}}    2018-05-16-01

Now I want to form another table with only 2 columns id and its respective score in it.
How can i do it in scala spark?

Moreover, whats really bugging me is how can I access only a particular score and how can I store it in an integer variable lets say temp?

can you provide with sample input, expected output and what you've tried? — Anahcolus
– Anahcolus, Commented May 22, 2018 at 9:59
@RameshMaharjan i have edited and added an example, i want to store that score 0.21278 inside a variable integer temp what to do? and also i also want to create a new table which contains those all ids and scores. please help — Romal Jaiswal
– Romal Jaiswal, Commented May 22, 2018 at 10:07
can you format the input data according to the table format? — Anahcolus
– Anahcolus, Commented May 22, 2018 at 10:10
@RameshMaharjan its actually formatted according to the table format , 1st line of sample row indicated id , 2nd row indicates the persona_model, and the 3rd row indicates process_date — Romal Jaiswal
– Romal Jaiswal, Commented May 22, 2018 at 10:12

illak zapata · Accepted Answer · 2018-05-23 16:11:09Z

1

You can do this:

val newDF = oldDF.select(col("id"), col("persona_model")("Tech")("score").as("temp"))

then you can extract temp values easily.

update: if you have more than one Key then the procedure is a little more complex.

first create a class for the struct (necesary for type cast):

case class Score(score: Double, tag: String)

then extract all the keys from the data:

val keys = oldDF.rdd
    .flatMap(r => r.getMap(1).asInstanceOf[Map[String, Score]].toList)
    .collect.map(_._1).distinct.toList

finally you can extract all names like this:

def condition(keys: List[String]): Column = {
     keys match {
        case k::ks => when(col("persona_model")(k)("score").isNotNull, col("persona_model")(k)("score")).otherwise(condition(ks))
        case nil  => lit(null)
     }
 }

val newDF = oldDF.select(col("id"), condition(keys))

edited May 23, 2018 at 16:11

answered May 22, 2018 at 23:00

illak zapata

662 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Romal Jaiswal Over a year ago

holy shit, it worked, i love u illak , just 1 more thing , it creates a table but shows score value only for those whose map key is "Tech" and null for others, can u fix it please :)

illak zapata Over a year ago

updated answer, it works only if the map only has one element (one pair key-value)

Collectives™ on Stack Overflow

how to access map values and keys stored in a data frame in scala spark

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related