Using "snow" parallel operations in bootstrap_parameters/model on merMod object (R)

Question

I've been using bootstrap_parameters (parameters package in R) on generalised linear mixed models produced using glmmTMB. These work fine without parallel processing (parallel = "no") and also works fine on my old and slow mac using parallel = "multicore". I'm working on a new PC (Windows OS) so need to use parallel = "snow" however I get the following error:

system.time(b <- bootstrap_parameters(m1, iterations = 10, parallel = "snow", n_cpus = 6)) Error in data.frame(..., check.names = FALSE) : arguments imply differing number of rows: 0, 1 In addition: Warning message: In lme4::bootMer(model, boot_function, nsim = iterations, verbose = FALSE, : some bootstrap runs failed (10/10) Timing stopped at: 0.89 0.3 7.11

If I select n_cpus = 1, the function works or if I feed bootstrap_parameters or bootstrap_model an lm object (where the underlying code uses boot::boot) it also works fine. I have narrowed the problem down to bootMer (lme4). I suspect the dataset exported using clusterExport is landing in an environment that is different from where clustered bootMer function is looking. The following is a reproduceable example

library(glmmTMB)
library(parameters)
library(parallel)
library(lme4)

m1 <- glmmTMB(count ~ mined + (1|site), zi=~mined,
              family=poisson, data=Salamanders)
summary(m1)

cl <- makeCluster(6)
clusterEvalQ(cl, library("lme4"))
clusterExport(cl, varlist = c("Salamanders"))

system.time(b <- bootstrap_parameters(m1, iterations = 10, parallel = "snow", n_cpus = 6))

stopCluster(cl)

Any ideas on solving this problem?

This is very likely a bug, but it's going to be tricky because it doesn't appear to affect bootMer directly: bootMer(m1, FUN = \(x)fixef(x)$cond, nsim =5, cl = cl) works for me, for example. — Ben Bolker
– Ben Bolker, Commented Jul 22, 2022 at 1:48
Hi Ben, Thanks for getting back to me. I ran your bootMer line which worked however when I added the argument parallel = "snow" it fell apart. I think bootMer might have a bug. system.time(c <- bootMer(m1, FUN = \(x)fixef(x)$cond, nsim =24, ncpus = 6, cl = cl, parallel = "snow")) > user system elapsed 0.09 0.02 0.14 Warning message: In bootMer(m1, FUN = function(x) fixef(x)$cond, nsim = 24, ncpus = 6, : some bootstrap runs failed (24/24) — plants-22
– plants-22, Commented Jul 23, 2022 at 10:18

Ben Bolker · Accepted Answer · 2023-06-30 13:28:48Z

0

You need to clusterEvalQ(cl, library("glmmTMB")). From https://github.com/glmmTMB/glmmTMB/issues/843:

This issue is more or less resolved by a documentation patch (we need to explicitly clusterEvalQ(cl, library("glmmTMB"))). The only question is whether we can make this any easier for users. There are two problems here: (1) when the user sets up their own cluster rather than leaving it to be done in bootMer, more explicit clusterEvalQ/clusterExport stuff is necessary in any case; (2) bootMer internally does parallel::clusterExport(cl, varlist=getNamespaceExports("lme4")) if it is setting up the cluster (not if the cluster is set up and passed to bootMer by the user), but we wouldn't expect it to extend the same courtesy to glmmTMB ...

For example, this all works:

library(glmmTMB)
library(parameters)
library(parallel)
library(lme4)

m1 <- glmmTMB(count ~ mined + (1|site), zi=~mined,
              family=poisson, data=Salamanders)
summary(m1)

par_cores <- max(1, floor(detectCores()/2))
par_cluster <- makeCluster(rep("localhost", par_cores), outfile = "log.txt")

clusterEvalQ(par_cluster, library("lme4"))
clusterExport(par_cluster, varlist = c("Salamanders"))

system.time(b <- bootstrap_parameters(m1, iterations = 10, cluster = par_cluster))

system.time(b2 <- bootMer(m1,
                          FUN = function(x)fixef(x)$cond,
                          nsim = 10, cl = par_cluster))
stopCluster(par_cluster)

edited Jun 30, 2023 at 13:28

answered Oct 10, 2022 at 12:35

Ben Bolker

230k26 gold badges405 silver badges497 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Karthik Thrikkadeeri Over a year ago

I am facing the same issue, but unfortunately your suggestion doesn't help. I've tried clusterEvalQ(cl, library("glmmTMB")), clusterEvalQ(cl, library(glmmTMB)), and clusterEvalQ(cl, {library(lme4); library(glmmTMB)}), but none of them work. And OP has correctly narrowed down that parallel = "snow" and ncpus > 1 are the problem. Worth reopening the issue? NB: My steps to make the cluster are slightly different: par_cores <- max(1, floor(detectCores()/2)) par_cluster <- makeCluster(rep("localhost", par_cores), outfile = "log.txt") where par_cluster corresponds to cl.

Ben Bolker Over a year ago

Are you using bootstrap_model or bootMer? When you use clusterEvalQ are you using par_cluster rather than cl as the first argument?

Ben Bolker Over a year ago

See edits. If these don't help you, please post a new question (linking to this one).

Karthik Thrikkadeeri Over a year ago

Thanks, Ben, but this didn't work for me. I have posted a new question here.

Collectives™ on Stack Overflow

Using "snow" parallel operations in bootstrap_parameters/model on merMod object (R)

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related