Spark Dataframe, add new column with function using other columns

Question

In my scala program, I have a dataframe df with two columns a and b (both of type Int). Aside I have a previously defined object obj with some methods and attributes. Here I want to add a new column to my dataframe df using the current values of the dataframe and attributes from obj.

For example, if I have the dataframe below :

+---+---+
| a | b |
+---+---+
| 1 | 0 |
| 4 | 8 |
| 2 | 5 |
+---+---+

and if obj has an attribute num: Int = 10 as well as a method f(a: Int, b: Int): Int = {a + b - this.num}, I want to use f to create the new column c like so :

+---+---+-----+
| a | b |  c  |
+---+---+-----+
| 1 | 0 | -9  |
| 4 | 8 |  2  |
| 2 | 5 | -3  |
+---+---+-----+

So the idea is: for each row, take the values of columns a and b and call the method f on obj with a and b as arguments too get the value that we then store in the corresponding row of the new column c. I tried to do something like this :

df = df.withColumn("c", obj.f(col("a"), col("b")))

but obviously it doesn't work as col() return a column and not the elements of that column. I also tried a foreach on a new column filled with 0s to fill the column row by row, but it didn't work as well.

Do you see how I can achieve this in Scala ?

Thank you.

pasha701 · Accepted Answer · 2021-10-21 12:03:58Z

1

The same result can be achieved without function, performance will be better:

val num = 10
df.withColumn("c", col("a") + col("b") - lit(num))

Version with UDF:

val num = 10
val f = (a: Int, b: Int) => {a + b - num}
val fUDF = udf(f)
df.withColumn("c", fUDF(col("a"), col("b")))

edited Oct 21, 2021 at 12:03

answered Oct 21, 2021 at 10:01

pasha701

7,2171 gold badge17 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Karzyfox Over a year ago

Well this was just an exemple, I really have to pass by the fuction. In a more general case where I manipulate more complex types than Int, this won't work. I use Int in my example to make it simple

Karzyfox Over a year ago

Thanks for your edit, now it works fine ! I didn't thought that user defined function (UDF) could be use in that case. Thanks for the tip ;)

Collectives™ on Stack Overflow

Spark Dataframe, add new column with function using other columns

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related