Scala: How to use custom function in parallel

Question

This should be a quick one :)

Let's say I've got a function like this:

    def sort: List[Int] => List[Int]

and I want to parallelize it in a function called sortPar. Originally I'd thought that I'd need to do this:

    def sortPar: List[Int] => List[Int] = ls => sort(ls.par).toList

but of course that's not possible since sort expects a list rather than a ParSeq. After some time playing around with it I came up with this solution, but I'm not too sure about it:

    def sortPar: List[Int] => List[Int] = ls => sort(ls).par.toList

Does this achieve anything in terms of runtime? I get no red crosses in eclipse so I assume it should work, but I don't know if it actually sorts in parallel.

Many thanks Curtis

In general, not every function is easily parallelizable and therefore there is no standard way of turning a function parallel. You would have to implement sortPar on your own. — Conrad
– Conrad, Commented Jul 15, 2014 at 7:29
So my function would run sort, then turn the result into a parSeq and then back into a list? — Curtis
– Curtis, Commented Jul 15, 2014 at 7:35

Kigyo · Accepted Answer · 2014-07-15 10:20:19Z

def sortPar: List[Int] => List[Int] = ls => sort(ls).par.toList does not do anything in parallel, since you first call sort(ls), which sorts the list sequentially.

The calls after that are useless, because you turn the result into a parallel collection and directly translate it back to a sequential List.

Things that you can/should consider:

Regarding your first problem, there are traits that are designed for such cases, like GenMap, GenSet or in your case GenSeq might be the closest, since a List is a Seq.

So you can write your sort function like this:

def sort(seq: scala.collection.GenSeq[Int]) = ...

and use it with a linear or parallel Seq.

sort(List(1,2,3))
sort(List(1,2,3).par)

So if you implement your sort with functions that are available to GenSeq, passing a sequential Seq will use the sequential implementation and passing a ParSeq might run those functions in parallel. (Some functions still run sequentially, like sorted)

If you don't use functions that have a parallel implementation, then you have to resort to taking care of parallelization yourself. I personally have no experience with it. You can look at some implementations in the source code.

https://github.com/scala/scala/blob/2.12.x/src/library/scala/collection/parallel/ParIterableLike.scala

Collectives™ on Stack Overflow

Scala: How to use custom function in parallel

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related