GroupByKey with datasets in Spark 2.0 using Java

Question

I have a dataset containing data like the following:

|c1| c2|
---------
| 1 | a |
| 1 | b |
| 1 | c |
| 2 | a |
| 2 | b |

...

Now, I want to get the data grouped like the following (col1: String Key, col2: List):

| c1| c2 |
-----------
| 1 |a,b,c|
| 2 | a, b|
...

I thought that using goupByKey would be an sufficient solution, but I can't find any example, how to use it.

Can anyone help me to find a solution using groupByKey or using any other combination of transformations and actions to get this output by using datasets, not RDD?

abaghel · Accepted Answer · 2016-11-19 04:01:42Z

3

Here is Spark 2.0 and Java example with Dataset.

public class SparkSample {
    public static void main(String[] args) {
    //SparkSession
    SparkSession spark = SparkSession
            .builder()
            .appName("SparkSample")
            .config("spark.sql.warehouse.dir", "/file:C:/temp")
            .master("local")
            .getOrCreate();     
    //input data
    List<Tuple2<Integer,String>> inputList = new ArrayList<Tuple2<Integer,String>>();
    inputList.add(new Tuple2<Integer,String>(1, "a"));
    inputList.add(new Tuple2<Integer,String>(1, "b"));
    inputList.add(new Tuple2<Integer,String>(1, "c"));
    inputList.add(new Tuple2<Integer,String>(2, "a"));
    inputList.add(new Tuple2<Integer,String>(2, "b"));          
    //dataset
    Dataset<Row> dataSet = spark.createDataset(inputList, Encoders.tuple(Encoders.INT(), Encoders.STRING())).toDF("c1","c2");
    dataSet.show();     
    //groupBy and aggregate
    Dataset<Row> dataSet1 = dataSet.groupBy("c1").agg(org.apache.spark.sql.functions.collect_list("c2")).toDF("c1","c2");
    dataSet1.show();
    //stop
    spark.stop();
  }
}

answered Nov 19, 2016 at 4:01

abaghel

15.4k2 gold badges56 silver badges69 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

abaghel Over a year ago

Glad I could help.

J Bentz · Accepted Answer · 2016-11-18 19:27:41Z

1

With a DataFrame in Spark 2.0:

scala> val data = List((1, "a"), (1, "b"), (1, "c"), (2, "a"), (2, "b")).toDF("c1", "c2")
data: org.apache.spark.sql.DataFrame = [c1: int, c2: string]
scala> data.groupBy("c1").agg(collect_list("c2")).collect.foreach(println)
[1,WrappedArray(a, b, c)]
[2,WrappedArray(a, b)]

answered Nov 18, 2016 at 19:27

J Bentz

452 silver badges5 bronze badges

Comments

Allan · Accepted Answer · 2017-12-06 06:24:19Z

0

This will read the table in to dataset variable

Dataset<Row> datasetNew = dataset.groupBy("c1").agg(functions.collect_list("c2"));
datasetNew.show()

edited Dec 6, 2017 at 6:24

Allan

12.5k3 gold badges33 silver badges56 bronze badges

answered Dec 6, 2017 at 4:59

Vijay Anantharamu

313 bronze badges

Collectives™ on Stack Overflow

GroupByKey with datasets in Spark 2.0 using Java

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related