how to ascending sort a multiple array of SPARK RDD by any column in scala?

Question

I'm interested in apache SPARK.

I tried to ascending sort a multiple array of SPARK RDD by any column in scala.

(i.e. RDD[Array[Int] -> Array(Array(1,2,3), Array(2,3,4), Array(1,2,1))

If I sort by first column, then result will be Array(Array(1,2,3), Array(1,2,1), Array(2,3,4)). or If I sort by third column, then result will be Array(Array(1,2,3), Array(1,2,3), Array(2,3,4)). ) and then, I want to get RDD[Array[Int]] return-type value. Is there a method to solve it, whether using map() or filter() function?

Tzach Zohar · Accepted Answer · 2016-10-24 07:59:06Z

2

Use RDD.sortBy:

// sorting by second column (index = 1)
val result: RDD[Array[Int]] = rdd.sortBy(_(1), ascending = true)

The sorting function can also be written using Pattern Matching:

val result: RDD[Array[Int]] = rdd.sortBy( {
  case Array(a, b, c) => b /* choose column(s) to sort by */
}, ascending = true)

Also note the ascending argument's default value is true, so you can drop it and get the same result:

val result: RDD[Array[Int]] = rdd.sortBy(_(1))

answered Oct 24, 2016 at 7:59

Tzach Zohar

37.9k3 gold badges83 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

S.Kang Over a year ago

It works. Thank you, but could you answer one more question if you don't mind? If I insert 4 or more dimension arrays, then I have to command by typing different case-statement.

S.Kang Over a year ago

Never Mind about my last comment. Thank you~~

Balaji Reddy · Accepted Answer · 2016-10-24 09:57:28Z

0

val baseRdd = sc.parallelize(Array(Array(1, 2, 3), Array(2, 3, 4), Array(1, 2, 1)))

//False specifies desending order 
val result = baseRdd.sortBy(x => x(1), false)

result.foreach { x => println(x(0) + "\t" + x(1) + "\t" + x(2)) }

Result

2 3 4

1 2 3

1 2 1

answered Oct 24, 2016 at 9:57

Balaji Reddy

5,7103 gold badges38 silver badges48 bronze badges

6 Comments

S.Kang Over a year ago

Could you answer one more question if you don't mind? I want to make a new RDD array with some elements extracted from original RDD array. (i.e. Here is a RDD[Array[Int]] -> Array(Array(1,2,3), Array(2,3,4), Array(1,2,1). And I want to make a new RDD array from original RDD array -> Array(Array(1,2,3), Array(1,2,1)) like this.) Is there a method to solve it?

Balaji Reddy Over a year ago

Yes you can do that. one quick question. on what basis you want to extract data from ur original rdd ? that would help me to give u precise answer.

S.Kang Over a year ago

original rdd array means that RDD[Array[Int]] -> Array(Array(1,2,3), Array(2,3,4), Array(1,2,1)). As I say, I wanna make a new rdd array (Array(Array(1,2,3), Array(1,2,1)) from RDD[Array[Int]] -> Array(Array(1,2,3), Array(2,3,4), Array(1,2,1)).` It's difficult to me...

Balaji Reddy Over a year ago

you need to apply filter() method. filter method will return true or false. I'm not sure what are your parameters to filter it. but giving one sample ..

Balaji Reddy Over a year ago

val baseRdd = sc.parallelize(Array(Array(1, 2, 3), Array(2, 3, 4), Array(1, 2, 1))) val resultRDD = baseRdd.filter { x => x(1).!=(3) } resultRDD.foreach { x => println(x(0) + "\t" + x(1) + "\t" + x(2)) }

|

Collectives™ on Stack Overflow

how to ascending sort a multiple array of SPARK RDD by any column in scala?

2 Answers 2

2 Comments

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related