Spark SQL UDF returning scala immutable Map with df.WithColumn()

Question

I have case class

case class MyCaseClass(City : String, Extras : Map[String, String])

and user defined function which returns scala.collection.immutable.Map

def extrasUdf = spark.udf.register(
   "extras_udf", 
   (age : Int, name : String) => Map("age" -> age.toString, "name" -> name)
)

but this breaks with Exception:

import spark.implicits._

spark.read.options(...).load(...)
      .select('City, 'Age, 'Name)
      .withColumn("Extras", extrasUdf('Age, 'Name))
      .drop('Age)
      .drop('Name)
      .as[MyCaseClass]

I should use spark sql's MapType(DataTypes.StringType, DataTypes.IntegerType) but I can't find any working example...

And this works if I use scala.collection.Map but I need immutable Map

Sim · Accepted Answer · 2016-07-25 19:01:45Z

4

There are many problems with your code:

You are using def extrastUdf =, which creates a function for registering a UDF as opposed to actually creating/registering a UDF. Use val extrasUdf = instead.
You are mixing value types in your map (String and Int), which makes the map be Map[String, Any] as Any is the common superclass of String and Int. Spark does not support Any. You can do at least two things: (a) switch to using a string map (with age.toString, in which case you don't need a UDF as you can simply use map()) or (b) switch to using named structs using named_struct() (again, without the need for a UDF). As a rule, only write a UDF if you cannot do what you need to do with the existing functions. I prefer to look at the Hive documentation because the Spark docs are rather sparse.
Also, keep in mind that type specification in Spark schema (e.g., MapType) is completely different from Scala types (e.g., Map[_, _]) and separate from how types are represented internally and mapped between Scala & Spark data structures. In other words, this has nothing to do with mutable vs. immutable collections.

Hope this helps!

answered Jul 25, 2016 at 19:01

Sim

13.6k11 gold badges69 silver badges96 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

fpopic Over a year ago

Is it possible without/with UDF to put inside WithColumn() expression to put 'Age, 'Name in map only if they are not null cause I don't want to have null values in map?

Sim Over a year ago

You could use if() or a case when ... then expression.

Collectives™ on Stack Overflow

Spark SQL UDF returning scala immutable Map with df.WithColumn()

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related