Implementing IF condition in a map function spark scala

Question

val keyValue = parsedData.map(x => (x.col1,x.col2,
  if(x.col3 = "a") "0.1" 
  else if(x.col3 = "b") "0.2" 
  else if(x.col3 = "c") "0.3" 
  else if(x.col3 = "d") "0.4"     
  else if(x.col3 = "e") "0.5"))

My parsedData rdd has 3 columns and for my third column i am trying to do the above, and you might have guessed its not working.

Any ideas how to implement this ?

What error are you getting? This looks valid to me. The only thing I notice is that you'll end up with a Unit type for the last column since you have no else — Justin Pihony
– Justin Pihony, Commented Oct 21, 2015 at 14:45
Its saying expression of type unit doesn't conform to expected type boolean. parsedData has data like (1,2,a),(2,3,b),(3,4,c) — Achillies57
– Achillies57, Commented Oct 21, 2015 at 14:54
still doesn't answer my question, what is the type of x? can you run :type parsedData if you are using spark-shell? — eliasah
– eliasah, Commented Oct 21, 2015 at 14:58

Justin Pihony · Accepted Answer · 2015-10-21 15:11:51Z

1

There are a couple of problems:

You should be using == instead of =
You need an else or else your type for the 3rd item will be Unit...but that will still compile. Worst case, you should maybe use an Option?

answered Oct 21, 2015 at 15:11

Justin Pihony

67.2k20 gold badges154 silver badges185 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Daniel Darabos Over a year ago

You could also use a match to be more concise. x.col3 match { case "a" => "0.1" case "b" => "0.2" ... }

Collectives™ on Stack Overflow

Implementing IF condition in a map function spark scala

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related