How to dynamically create the list of the columns to include in select?

Question

I tried to "generate" a spark query in this way

  def stdizedOperationmode(sqLContext: SQLContext,withrul: DataFrame): DataFrame = {
    // see http://spark.apache.org/docs/latest/sql-programming-guide.html
    import sqLContext.implicits._
    val AZ: Column  = lit(0.00000001)

    def opMode(id:Int): Column = {
      (column("s"+id) - coalesce(column("a"+id) / column("sd"+id), column("a"+id) / lit(AZ))).as("std"+id)
    }

    // add the 21 std<i> columns based on s<i> - (a<id>/sd<id>)
    val columns: IndexedSeq[Column] = 1 to 21 map(id => opMode(id))

    val withStd = withrul.select(columns:_*)

    withStd
  }

Question how do I add "all other columns" (*) idea: something like withrul.select('* :+ columns:_*)

zero323 · Accepted Answer · 2016-04-13 17:51:54Z

4

You can try the following :

// add the 21 std<i> columns based on s<i> - (a<id>/sd<id>)

val columns: IndexedSeq[Column] = 1 to 21 map(id => opMode(id))

val selectAll: Array[Column] = (for {
  i <- withrul.columns
} yield withrul(i)) union columns.toSeq

val withStd = withrul.select(selectAll :_*)

The second line will yeild all the columns from withrul adding them with column as a Seq[Column]

You are not obliged to create a value to return it afterward, can replace the last 2 lines with :

withrul.select(selectAll : _*)

edited Apr 13, 2016 at 17:51

zero323

331k108 gold badges982 silver badges958 bronze badges

answered Apr 11, 2016 at 9:54

eliasah

40.5k12 gold badges128 silver badges159 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Jacek Laskowski Over a year ago

I'd replace for...yield to union later with map. I'd also recommend (1 to 21).map(opMode).

Collectives™ on Stack Overflow

How to dynamically create the list of the columns to include in select?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related