How to sort dataframe in Spark without using Spark SQL ?

Question

I'm working with Spark now but I find out that using ORDER BY in Spark SQL is very slow to sort a DataFrame. So how to sort a DataFrame without Spark SQL ?

Fabio Fantoni · Accepted Answer · 2015-04-20 07:37:23Z

1

I'm not sure if I've fully understand what you need.

Anyway, if you want to sort a DF you could use sortBy (or sortByKey in case of (K,V))

For example, if we assume to have a DF (in this case coming from Spark SQL), we can sort it like this:

val sqlResult = sqlContext.sql("select first_column, second_column from logs").toDF()
val result = sqlResult.sortBy(x=>x._1) // first column sorting

As said before, you can sort any DF, but I just want to show another way to "access" data with Spark SQL, and then sorting them with Spark core functionalities.

Hope it could help!

FF

answered Apr 20, 2015 at 7:37

Fabio Fantoni

3,1573 gold badges26 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Fabio Fantoni Over a year ago

If I helped you, could you please rate and accept the answer? have a good day!

Collectives™ on Stack Overflow

How to sort dataframe in Spark without using Spark SQL ?

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related