Converting Python to Scala in Spark ML?

Question

The question is about Logistic regression with spark ml (data frames)

When I want to change the code Python to Scala

Python:

[stage.coefficients for stage in model.stages
    if isinstance(stage, LogisticRegressionModel)]

Scala:(changed)

   for (stage<-model.stages){
        if(stage.isInstanceOf[LogisticRegressionModel]{
            val a = Array(stage.coefficients)
    }}

I have already checked stage.isInstanceOf[LogisticRegressionModel], which returned the True. However, stage.coefficients has the error message. It says "value coefficients is not a member of org.apache.spark.ml.Transformer".

I only check the stage, it will return

org.apache.spark.ml.Transformer= logreg 382456482

Why the type is different when the isInstanceOf returns true? What should I do? Thanks

zero323 · Accepted Answer · 2016-08-02 17:17:16Z

2

Why the type is different when the isInstanceOf returns true?

Well, Scala is a statically typed language and stages is an Array[Transformer] so each element you access is a Transformer. Transformers in general have no coefficients, hence the error.

What should I do?

Be specific about the types.

import org.apache.spark.ml.classification.LogisticRegressionModel

model.stages.collect { 
  case lr: LogisticRegressionModel => lr.coefficients
}.headOption

answered Aug 2, 2016 at 17:17

zero323

331k108 gold badges982 silver badges958 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Martin TT Over a year ago

Thanks. I want to ask one more question. It returns Some. How can I convert Some to Array?

zero323 Over a year ago

It returns an Option[o.a.s.mllib.linalg.Vector]. If you're sure that required stage exists you could simply use head (instead of headOption) and o.a.s.mllib.linalg.Vector has toArray method.

Collectives™ on Stack Overflow

Converting Python to Scala in Spark ML?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related