make .combine function scaleable

Question

I am trying to use foreach and am having problems making the .combine function scalable. For example, here is a simple combine function

MyComb <- function(part1,part2){
          xs <- c(part1$x,part2$x)
          ys <- c(part1$y,part2$y)
          return(list(xs,ys))
          }

When I use this function to combine a foreach statement with an iterator other than 2 it returns it incorrectly. For example this works:

   x = foreach(i=1:2,.combine=MyComb) %dopar% list("x"=i*2,"y"=i*3)

But not this:

 x = foreach(i=1:3,.combine=MyComb) %dopar% list("x"=i*2,"y"=i*3)

Is there a way to generalize the combine function to make it scalable to n iterations?

Brian Diggs · Accepted Answer · 2011-09-26 22:45:43Z

3

Your .combine function must take either two pieces and return something that "looks" like a piece (could be passed back in as a part) or take many arguments and put all of them together at once (with the same restrictions). Thus at least your MyComb must return a list with components x and y (which is what each piece of your %dopar% do.

A couple of ways to do this:

MyComb1 <- function(part1, part2) {
    list(x=c(part1$x, part2$x), y=c(part1$y, part2$y))
}

x = foreach(i=1:3,.combine=MyComb1) %dopar% list("x"=i*2,"y"=i*3)

This version takes only two pieces at a time.

MyComb2 <- function(...) {
    dots = list(...)
    ret <- lapply(names(dots[[1]]), function(e) {
        unlist(sapply(dots, '[[', e))
    })
    names(ret) <- names(dots[[1]])
    ret
}

s = foreach(i=1:3,.combine=MyComb2) %dopar% list("x"=i*2,"y"=i*3)
x = foreach(i=1:3,.combine=MyComb2, .multicombine=TRUE) %dopar% list("x"=i*2,"y"=i*3)

This one can take multiple pieces at a time and combine them. It is more general (but more complex).

answered Sep 26, 2011 at 22:45

Brian Diggs

59.1k14 gold badges169 silver badges189 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

scottyaz Over a year ago

Thanks! So in my actual application each bit supplied to MyComb is a list with a bunch of elements each being matrix. I want to abind(...,along=3) each matrix of the same name from the original list and finally return a big list with these arrays. I am having trouble seeing how to do that with your MyComb2 solution. Any ideas?

Brian Diggs Over a year ago

@scottyaz Probably just replace unlist(...) with abind(...,along=3), but without building a working example, I am not sure. The result of sapply(dots, '[[', e) is a list of the components with the name "e" from every part. abind can take a list as its argument, so it should work with what is returned from the sapply.

Collectives™ on Stack Overflow

make .combine function scaleable

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related