Apply UDF dynamically to 1 to N columns from N columns in dataframe

Question

I have a schema and name of columns to apply UDF to. Name of columns are user input and they can differ in numbers for each input. Is there a way to apply UDFs to N columns in dataframe ?

Trying to achieve this. for schema with say col1,col2,col3,col4,col5

  DataFrame newDF = df.withColumn("col2", callUDF("test", (df.col("col2"))));
  or 
  DataFrame newDF = df.withColumn("col2", callUDF("test", (df.col("col2"))))
                 .withColumn("col3", callUDF("test", (df.col("col3"))));
  or
   DataFrame newDF = df.withColumn("col2", callUDF("test", (df.col("col1"))))
                 .withColumn("col3", callUDF("test", (df.col("col3"))))
                 .withColumn("col5", callUDF("test", (df.col("col5"))))
  or for N columns.

Any ideas ?

Check this stackoverflow.com/questions/34037889/…

Indrajit Swain
– Indrajit Swain

2017-03-15 06:24:00 +00:00
Commented Mar 15, 2017 at 6:24 — Indrajit Swain
– Indrajit Swain, Commented Mar 15, 2017 at 6:24

anup · Accepted Answer · 2017-03-30 22:26:19Z

0

I ended up writing code to dynamically generate SPARK SQL query for applying UDFs to 1 to N cols. Then register input dataframe as temp table and use genererated query.

answered Mar 30, 2017 at 22:26

anup

112 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Giridhar Karnik Over a year ago

Care to share the code? Or else your answer is of no use.

Collectives™ on Stack Overflow

Apply UDF dynamically to 1 to N columns from N columns in dataframe

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related