Error while calling udf from within withColumn in Spark using Scala

Question

I receive and error while calling udf from within withColumn in Spark using Scala. This error happens while building with SBT.

val hiveRDD = sqlContext.sql("select * from iac_trinity.ctg_us_clickstream")
hiveRDD.persist()

val trnEventDf = hiveRDD
  .withColumn("system_generated_id", getAuthId(hiveRDD("session_user_id")))
  .withColumn("application_assigned_event_id", hiveRDD("event_event_id"))


val getAuthId = udf((session_user_id:String) => {
    if (session_user_id != None){
        if (session_user_id != "NULL"){
            if (session_user_id != "null"){
            session_user_id
          }else "-1"
        }else "-1"
    }else "-1"
  }

)

I receive the error which is -

scala:58: No TypeTag available for String
val getAuthId = udf((session_user_id:String) => {

It compiles properly when instead of (session_user_id:String) I use (session_user_id:Any) but fails in runtime as Any is not recognized in Spark. Please let me know how to handle this.

Error or not this doesn't make sense. Object of class String cannot be None! — zero323
– zero323, Commented Jul 4, 2016 at 8:14

Justin Pihony · Accepted Answer · 2016-07-04 03:12:39Z

1

Have you tried being explicit with your types?

udf[String, String]((session_user_id:String)...

answered Jul 4, 2016 at 3:12

Justin Pihony

67.2k20 gold badges154 silver badges185 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

preitam ojha Over a year ago

Yes , I have tried being explicit - val getAuthId = udf[String,String]((session_user_id:String) => if (session_user_id == None) .... the error is the same - scala:57: No TypeTag available for String [error] val getAuthId = udf[String,String]((session_user_id:String) => if (session_user_id == None)"-1"

eliasah Over a year ago

@preitamojha are you sure you are executing the same code your are giving us ? It seems unlikely that this doesn't work. I can't reproduce the error.

Collectives™ on Stack Overflow

Error while calling udf from within withColumn in Spark using Scala

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related