Spark Scala: Convert Map into Row object

Question

I want to convert a Scala map into Row object (basically what Row(**dict) does in python I have to achieve it in Scala spark).

input : Map(com.project.name -> "A", com.project.age -> 23 )
output : Row(com.project.name="A", com.project.age = 23)

Please help.

@Dipali_Deshmukh Do you want it as dataframe or as Row? Also want are you trying to do here with this Row? — Nikunj Kakadiya
– Nikunj Kakadiya, Commented Nov 25, 2020 at 13:48
Row object , If we cannot do that then Can we convert it into dataframe and then from that dataframe we get Row object ? Row should return ultimately according to requirement, — Dipali Deshmukh
– Dipali Deshmukh, Commented Nov 25, 2020 at 13:52
you can check my answer below if that is helpful or near to what you were looking for — Nikunj Kakadiya
– Nikunj Kakadiya, Commented Nov 25, 2020 at 14:33

Raphael Roth · Accepted Answer · 2020-11-26 19:22:37Z

3

You can use Row.fromSeq:

val m = Map("com.project.name" -> "A", "com.project.age" -> "23")
val row = Row.fromSeq(m.toSeq)

or alternatively Row(m.toSeq:_*)

both giving [(com.project.name,A),(com.project.age,23)]

answered Nov 26, 2020 at 19:22

Raphael Roth

27.3k19 gold badges98 silver badges152 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Nikunj Kakadiya · Accepted Answer · 2020-11-25 14:00:07Z

0

You can convert map into the dataframe as follows :

import org.apache.spark.sql.types._
import org.apache.spark.sql.functions._
val input : Map[String,String] = Map("com.project.name" -> "A", "com.project.age" -> "23")
val df = input.tail
  .foldLeft(Seq(input.head._2).toDF(input.head._1))((acc,curr) => 
acc.withColumn(curr._1,lit(curr._2)))

Now if you want to get the Row from the Dataframe you can get as follows :

val row = df.first

And if you want to see the names of the column you can get that as follows :

val columns = df.columns

answered Nov 25, 2020 at 14:00

Nikunj Kakadiya

3,0762 gold badges26 silver badges45 bronze badges

2 Comments

Dipali Deshmukh Over a year ago

Hi I really appreciate your response , But I want output as -> Row(com.project.name="A", com.project.age = 23).

Nikunj Kakadiya Over a year ago

if you want what you are looking for. you can utilize Map for that purpose as Map is itself collection of key-value pairs. Map is similar to dictionary

Collectives™ on Stack Overflow

Spark Scala: Convert Map into Row object

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related