Pass dataframe column name as parameter to the function using scala?

Question


I have a function that adds 2 columns:

def sum_num (num1: Int, num2: Int): Int = {
    return num1 + num2
}

I have a dataframe df with below values

+----+----+----+
|col1|col2|col3|
+----+----+----+
|1   |2   |5   |
|7   |4   |4   |
+----+----+----+

I want to add a column and pass column names to the function but the below code is not working. It gives error found Column required is Int

val newdf = df.withColumn("sum_of_cols1", sum_num($col1, $ col2))
              .withColumn("sum_of_cols2", sum_num($col1, $ col3))

@GuruStron I had seen this but not sure how to create the udf using multiple columns. Also i had read that udf might have some performance implications and since I am doing the calculation on billion records trying to find other solutions! — justanothertekguy
– justanothertekguy, Commented Mar 10, 2021 at 6:51

Emiliano Martinez · Accepted Answer · 2021-03-10 08:50:07Z

4

Change your code to:

import spark.implicits._

def sum_num (num1: Column, num2: Column): Column = {
  return num1 + num2
}

val newdf = df.withColumn("sum_of_cols1", sum_num($"col1", $"col2"))
  .withColumn("sum_of_cols2", sum_num($"col1", $"col3"))

You must operate over Spark SQL columns. You can do arithmetic operations with them. Take a look to the operators that can be used

edited Mar 10, 2021 at 8:50

answered Mar 9, 2021 at 21:11

Emiliano Martinez

4,1432 gold badges13 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

justanothertekguy Over a year ago

thanks for this. for function sum_num does the output need to be a column or can i out Int there since I need that value to do some other calculation

Emiliano Martinez Over a year ago

you can operate with column directly, using the available operators. Don´t worry about that, if you need other specific functions you could add an udf that takes as input a column too.

Collectives™ on Stack Overflow

Pass dataframe column name as parameter to the function using scala?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related