Running into pyspark.sql.utils.AnalysisException: UDF class doesn't implement any UDF interface

Question

I am trying to utilize Scala udfs from pyspark and running into 'pyspark.sql.utils.AnalysisException: UDF class doesn't implement any UDF interface' error

My scala code looks something like this

package com.spark.udfexample

Object test{
  case class (a: Option[String], b: Option[String])
   
    val foo: StructType = new StructType().add("a", StringType).add("b", StringType)

    def udf1:UserDefinedFunction = udf((param: String, param: Seq[Row])) {
       ...
       ...
    }
    ..
.
}

I am packing this object class as jar and passing to pyspark context.

spark.udf.registerJavaFunction("udf1", "com.spark.udfexample.test") - running into the error here, can someone help? thx

Chris · Accepted Answer · 2024-09-27 10:04:59Z

0

I'd suggest looking at this blog post as it gives a run down of why you are getting that error.

Your pseudo code looks to be a setup of scala udf potentially even using class state and what the python interface is looking for is a no args public class implementing a UDFx interface.

If you can successfully register your actual scala udf function (and use it) within scala spark then you can create a function in an object to register it via scala and call that in your python code.

answered Sep 27, 2024 at 10:04

Chris

2,96516 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Running into pyspark.sql.utils.AnalysisException: UDF class doesn't implement any UDF interface

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related