From what I have seen, in order to do this you have to
udf as a plain functionregister the function with SQLContext for SQL
spark.sqlContext.udf.register("myUDF", myFunc)
turn this into a UserDefinedFunction for DataFrame
def myUDF = udf(myFunc)
Is there no way to combine this into one step and make the udf available for both? Also, for cases where a function exists for DataFrame but not for SQL, how do you go about registering it without copying over the code again?
UDFRegistration.register variants, which take a scala.FunctionN, return an UserDefinedFunction so you can register SQL function and create DSL friendly UDF in a single step:
val timesTwoUDF = spark.udf.register("timesTwo", (x: Int) => x * 2)
spark.sql("SELECT timesTwo(1)").show
+---------------+
|UDF:timesTwo(1)|
+---------------+
| 2|
+---------------+
spark.range(1, 2).toDF("x").select(timesTwoUDF($"x")).show
+------+
|UDF(x)|
+------+
| 2|
+------+
You can use the following and still apply it on dataframe
spark.sqlContext.udf.register("myUDF", myFunc)
Use selectExpr when calling it on dataframe transformations.
df.selectExpr("myUDF(col1) as modified_col1")
Update for Spark2-
spark.udf.register("func_name", func_name)
Argument1- Function name it will be register in spark
Argument2- Function name what is defined while creating in python/scala
It's best practice to register the function with same name in spark.
