Spark/Scala - Failed to execute user defined function

Question

I have the following code that works.

val locList = Source.fromInputStream(getClass.getResourceAsStream("/locations.txt")).getLines().toList

def locCheck(col: String): Boolean = locList.contains(col)
def locUDF = udf[Boolean, String](locCheck)

But when I add a toUpperCase to make it

val locList = Source.fromInputStream(getClass.getResourceAsStream("/locations.txt")).getLines().toList

def locCheck(col: String): Boolean = locList.contains(col.toUpperCase)
def locUDF = udf[Boolean, String](locCheck)

I run into a Failed to execute user defined function caused by java.lang.NullPointerException

I using the udf as df.filter(locUDF('location)).count()

What am I doing wrong here and how do I fix it ?

Make sure that you don't have null values in location column if location is null then col is null and throws NPE in col.toUpperCase — koiralo
– koiralo, Commented Jun 22, 2020 at 13:11

koiralo · Accepted Answer · 2020-06-22 13:25:38Z

1

There is nothing wrong with the function or udf. The problem is with the data that comes into the udf.

Here in your case if the column location have a null values, When you pass those values to udf the value of col is null.

Then You get a NullPointerException when you call col.toUpperCase in case col is null.

You can simply check the null values in function

def locCheck(col: String): Boolean = if (col == null) false else locList.contains(col.toUpperCase)

Or you can use Options to handle this as

def locCheck(col: String): Boolean =locList.contains(Option(col).map(_.toUpperCase))

Hope this helps!

edited Jun 22, 2020 at 13:25

answered Jun 22, 2020 at 13:20

koiralo

23.2k6 gold badges57 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Ajay Over a year ago

Yes, all the columns seem to have the same issue

Ajay Over a year ago

Fixed it, the source data had some non-ascii characters which was affecting the inputStream. Once that was fixed, you solution for handling null worked great. Thank you !

Collectives™ on Stack Overflow

Spark/Scala - Failed to execute user defined function

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related