Adding JARs to use sql UDF in spark

Question

I want to use custom UDF which are defined in my libaray. I have used the following code for that:

%spark2
import org.apache.spark.sql.functions.year

val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)

sqlContext.sql(s"ADD JAR /usr/hdp/current/spark-client/lib/myLib.jar")

val df = sqlContext.sql("select parse_datetime(start_timestamp, 'CET', 'yyyyMMddHHmmssSSS') AS TIMESTAMP) from temp)

The above code complains about missing function "parse_datetime", so apparently, ADD JAR statement is not adding myLib. Is there a syntax problem or what is that I am missing. I am running this in Zeppelin.

I have also copied the myLib.jar in $SPARK_HOME/lib folder but no use.

wasif · Accepted Answer · 2017-04-14 09:06:01Z

1

I have found the solution. I have given the exact path of myLib.jar in the dependencies section of spark intertreper. It works now.

answered Apr 14, 2017 at 9:06

wasif

1051 gold badge4 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Vikash Pareek Over a year ago

How you have added jar to spark interpreter? Do you mean, adding jar in pom.xml or build.sbt file?

wasif Over a year ago

This was specific to Zeppelin. There one can add customer jars in the dependencies section of spark intertreper in Zepplin. What is that you are trying to do?

Collectives™ on Stack Overflow

Adding JARs to use sql UDF in spark

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related