How to call another object within Pyspark UDF function

Question

I have a class Hello with a few methods

I would like to create a hello object within a UDF pyspark function, such as:

def foo_generation(query_params):
  query_obj = Hello()
  foo = query_obj.hello_method(query_params)
  return foo

spark.udf.register("foo_generation", foo_generation)
df = df.withColumn("foo", F.expr("foo_generation(query_param_ES)"))

This doesn't appear to be working. How should I generate a Hello object in this instance?

chá de boldo · Accepted Answer · 2022-10-08 08:59:14Z

0

from pyspark.sql import SparkSession
from pyspark.sql.types import StringType
from pyspark.sql.functions import udf

@udf(returnType=StringType())
def foo_generation(str):
  query_obj = Hello()
  foo = query_obj.hello_method(str)
  return foo

df = df.withColumn("foo", F.expr("foo_generation(str)"))

answered Oct 8, 2022 at 8:59

chá de boldo

563 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to call another object within Pyspark UDF function

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related