How could I call a User defined function from spark sql queries in pyspark?

Question

I need to call a function from my spark sql queries. I have tried udf but I don't know how to manipulate it. Here is the scenario:

# my python function example

def sum(effdate, trandate):
  sum=effdate+trandate
  return sum

and my spark sql query is like:

spark.sql("select sum(cm.effdate, cm.trandate)as totalsum, name from CMLEdG cm ....").show()

These lines are not my code but I am stating it as an example. How could I call my sum function inside spark.sql(sql queries) for getting a result? Could you please kindly suggest me any link or any comment compatible with pyspark?

Any help would be appreciated.

Thanks

Kalyan

Hardik Desai · Accepted Answer · 2018-06-27 14:30:53Z

5

You just need to register your function as UDF:

from spark.sql.types import IntegerType()

# my python function example
def sum(effdate, trandate):
  sum=effdate+trandate
  return sum

spark.udf("sum", sum, IntegerType())
spark.sql("select sum(cm.effdate, cm.trandate)as totalsum, name from CMLEdG cm....").show()

edited Jun 27, 2018 at 14:30

user1143634

answered Jun 27, 2018 at 14:13

Hardik Desai

511 silver badge1 bronze badge

Sign up to request clarification or add additional context in comments.

1 Comment

Aris F. Over a year ago

spark.udf.register() worked for me (pyspark v3.3.1).

Arunakiran Nulu · Accepted Answer · 2016-10-04 09:01:41Z

3

Check this

    >>> from pyspark.sql.types import IntegerType
    >>> sqlContext.udf.register("stringLengthInt", lambda x: len(x), IntegerType())
    >>> sqlContext.sql("SELECT stringLengthInt('test')").collect()
    [Row(_c0=4)]

answered Oct 4, 2016 at 9:01

Arunakiran Nulu

2,0991 gold badge12 silver badges16 bronze badges

Collectives™ on Stack Overflow

How could I call a User defined function from spark sql queries in pyspark?

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related