Pyspark UDF using Quantlib function

Question

I have been experimenting with Quantlib and Spark, trying to pass a Quantlib function in Pyspark see example below:

from QuantLib import *
from pyspark.sql.types import StringType
from pyspark.sql.functions import udf


df = sc.parallelize([("2016-10-01",),
                     ("2016-11-01",),
                     ("2016-12-01",)]).toDF(['someDate'])

testudf = udf(lambda x: str(DateParser.parseFormatted(x,'%Y-%m-%d')), StringType())

df.withColumn('new', testudf('someDate')).show()

I haven't been successful so far and was wondering if anybody has had better luck.

Here is the error I get:

typeError: in method 'DateParser_parseFormatted', argument 1 of type 'std::string const &'
    at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
    at org.apache.spark.api.python.PythonRunner$$anon$1.<init>(PythonRDD.scala:234)

What's the exact type of the x that gets passed to the lambda inside udf? Is it a Python string, or some Spark type? — Luigi Ballabio
– Luigi Ballabio, Commented Feb 14, 2017 at 19:13

Luigi Ballabio · Accepted Answer · 2017-02-14 22:34:25Z

1

Being exported from C++, the DateParser.parseFormatted method is kind of particular about types and can't take the Spark string x that the udf machinery is passing to the lambda. You'll have to convert x back to a Python string inside the lambda. I'm not familiar with Spark and its types, but maybe str(x), as in

lambda x: str(DateParser.parseFormatted(str(x), '%Y-%m-%d'))

might do the work?

As a side note, I'm not sure what's the point of the outer str in your lambda. You're taking a string, converting it to a Date object by means of DateParser, and then you're converting the result to a string again...

answered Feb 14, 2017 at 22:34

Luigi Ballabio

4,44324 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pyspark UDF using Quantlib function

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related