Scala functional programming dry run

Question

Could you please help me in understanding the following method:

def extractGlobalID(custDimIndex :Int)(gaData:DataFrame) : DataFrame = {
  val getGlobId = udf[String,Seq[GenericRowWithSchema]](genArr => {
    val globId: List[String] =
      genArr.toList
        .filter(_(0) == custDimIndex)
         .map(custDim => custDim(1).toString)

    globId match {
      case Nil => ""
      case x :: _ => x
    }
  })

  gaData.withColumn("globalId", getGlobId('customDimensions))
}

Is it just me or is this rather poor code? Surely collectFirst followed by fold would be cleaner and faster? — Tim
– Tim, Commented Feb 8, 2019 at 18:29

Raphael Roth · Accepted Answer · 2019-02-08 20:49:55Z

1

The method applies an UDF to to dataframe. The UDF seems intended to extract a single ID from column of type array<struct>, where the first element of the struct is an index, the second one an ID.

You could rewrite the code to be more readable:

def extractGlobalID(custDimIndex :Int)(gaData:DataFrame) : DataFrame = {
  val getGlobId = udf((genArr : Seq[Row]) => {
    genArr
      .find(_(0) == custDimIndex)
      .map(_(1).toString)
      .getOrElse("")
  })

  gaData.withColumn("globalId", getGlobId('customDimensions))
}

or even shorter with collectFirst:

def extractGlobalID(custDimIndex :Int)(gaData:DataFrame) : DataFrame = {
  val getGlobId = udf((genArr : Seq[Row]) => {
    genArr
      .collectFirst{case r if(r.getInt(0)==custDimIndex) => r.getString(1)}     
      .getOrElse("")
  })

  gaData.withColumn("globalId", getGlobId('customDimensions))
}

edited Feb 8, 2019 at 20:49

answered Feb 8, 2019 at 19:26

Raphael Roth

27.3k19 gold badges98 silver badges152 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Luis Miguel Mejía Suárez Over a year ago

@Jay1991 if this solved your problem, you may accept it - see What should I do when someone answers my question?.

Collectives™ on Stack Overflow

Scala functional programming dry run

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related