How to pass the non dataframe argument to udf in spark

Question

Can any please help to know how to pass the non dataframe value as an argument to udf.

val df2 = df1.withColumn("newcol", udffunc(df1("col1"), x)).

The udf function i defined as below.

udffunc = udf(method _)

Method i defined as below.

def method(inputvar1: String, inputvar2: String): Option[Long] = {
  ...
  ...
  return Longvariable
}

Compile error. Its not allowing to pass the argument x as a argument. Can any one please help. — Ramesh
– Ramesh, Commented Sep 28, 2016 at 18:00
As per online looks like using withColumn there is no direct way to pass the external argument like hardcoding some string like I said. — Ramesh
– Ramesh, Commented Sep 29, 2016 at 3:02

koiralo · Accepted Answer · 2017-08-07 13:54:26Z

1

I am able to fix the issue by just writing the below code for UDF.

def udffunc = udf((y:String) => {
    val format:String = "x"
    method(y, "x")
  })

So with this, I am able to pass one argument(dataframe column) from .withColumn to the UDF function udffunc and then am able to pass external argument which is "x" in the above example to the function named as "method".

edited Aug 7, 2017 at 13:54

koiralo

23.2k6 gold badges57 silver badges77 bronze badges

answered Sep 29, 2016 at 4:28

Ramesh

1,5939 gold badges27 silver badges39 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Reactormonk Over a year ago

I'd recommend to always use val instead of var whenever possible. Immutable code is robust code.

Ramesh Over a year ago

Hi, using lit function i can pass the non dataframe argument from withColumn. Here is the code. val df2 = df1.withColumn(df1("col1"), lit(x))

Collectives™ on Stack Overflow

How to pass the non dataframe argument to udf in spark

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related