Adding a count column to my sequence in Scala

Question

Given the code below, how would I go about adding a count column? (e.g. .count("*").as("count"))

Final output to look like something like this:

+---+------+------+-----------------------------+------
| id|sum(d)|max(b)|concat_ws(,, collect_list(s))|count|
+---+------+------+-----------------------------+------
|  1|   1.0|  true|                          a. | 1 |
|  2|   4.0|  true|                          b,b| 2 |
|  3|   3.0|  true|                          c. | 1 |

Current code is below:

val df =Seq(
  (1, 1.0, true, "a"),
  (2, 2.0, false, "b")
  (3, 3.0, false, "b")
  (2, 2.0, false, "c")
).toDF("id","d","b","s")

val dataTypes: Map[String, DataType] = df.schema.map(sf => (sf.name,sf.dataType)).toMap

def genericAgg(c:String) = {
  dataTypes(c) match {
    case DoubleType => sum(col(c))
    case StringType => concat_ws(",",collect_list(col(c))) // "append"
    case BooleanType => max(col(c))
  }
}

val aggExprs: Seq[Column] = df.columns.filterNot(_=="id")
.map(c => genericAgg(c))

df
.groupBy("id")
.agg(aggExprs.head,aggExprs.tail:_*)
  .show()

Leo C · Accepted Answer · 2019-03-18 02:40:50Z

1

You can simply append count("*").as("count") to aggExprs.tail in your agg, as shown below:

df.
  groupBy("id").agg(aggExprs.head, aggExprs.tail :+ count("*").as("count"): _*).
  show
// +---+------+------+-----------------------------+-----+
// | id|sum(d)|max(b)|concat_ws(,, collect_list(s))|count|
// +---+------+------+-----------------------------+-----+
// |  1|   1.0|  true|                            a|    1|
// |  3|   3.0| false|                            b|    1|
// |  2|   4.0| false|                          b,c|    2|
// +---+------+------+-----------------------------+-----+

answered Mar 18, 2019 at 2:40

Leo C

22.5k3 gold badges28 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Adding a count column to my sequence in Scala

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related