Scala DataFrame, copy value of non null column into a new column

Question

I have a Scala DataFrame and want to add a new column to it such that its value is based on any of the two columns which is non null. E.g. in the below table, col1 and col2 are the columns that I want to add the new column based off of, such that if col1 is null then the new column has value from col2 and vice versa for col2. One of col1 and col2 will always have a non null value. I need to do this only in Scala without using SQL. Thank you for any help in advance.

<table><tbody><tr><th>Col1</th><th>Col2</th><th>Col3</th></tr><tr><td>11</td><td>null</td><td>Value13</td></tr><tr><td>null</td><td>22</td><td>Value23</td></tr><tr><td>32</td><td>32</td><td>Value33</td></tr></tbody></table>

This is the best I could try but it won't work.

df.withColumn("Col", df.Col1.isNull() ? df.Col2: df.Col1)

I found the answer here: stackoverflow.com/questions/43853949/… — SneakyPigeon
– SneakyPigeon, Commented Oct 12, 2020 at 7:37

Vladislav Varslavans · Accepted Answer · 2020-10-12 07:36:02Z

3

Try using when function:

df.withColumn("Col", when(col("Col1").isNull(), col("Col2")).otherwise(col("Col1")))

answered Oct 12, 2020 at 7:36

Vladislav Varslavans

2,9944 gold badges22 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

SneakyPigeon Over a year ago

Thanks Vladislav, I just found the answer on another post but have upvoted your answer :) It won't show though as I don't have enough rep.

Collectives™ on Stack Overflow

Scala DataFrame, copy value of non null column into a new column

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related