Assignment within Spark Scala foreach Loop

Question

I'm new to scala/spark and am trying to loop through a dataframe and assign the results as the loop progresses. The following code works but can only print the results to screen.

traincategory.columns.foreach { x=>

val test1 = traincategory.select("Id", x)

import org.apache.spark.ml.feature.{OneHotEncoder, StringIndexer}

//CODE TO PERFORM ONEHOT TRANSFORMATION

val encoded = encoder.transform(indexed)

encoded.show()

}

As val is immutable I have attempted to append the vectors from this transformation onto another variable, as might be done in R.

//var ended = traincategory.withColumn(x,encoded(0))

I suspect Scala has a more idiomatic way of processing this.

Thank you in advance for your help.

Have you looked at the Spark documentation ? spark.apache.org/docs/2.1.0/ml-features.html#onehotencoder — Marie
– Marie, Commented Apr 27, 2017 at 15:41

Clem Manger · Accepted Answer · 2017-04-28 15:33:03Z

0

A solution was available at :

https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/mllib/Correlations.scala

If anyone has similar issues with Scala MLIB there is great example code at :

https://github.com/apache/spark/tree/master/examples/src/main/scala/org/apache/spark/examples/mllib

answered Apr 28, 2017 at 15:33

Clem Manger

1931 silver badge13 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Assignment within Spark Scala foreach Loop

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related