2

I am using the atakrig package. When setting the number of cores to 8 for parallel computation I am getting this error: 0%Error in serialize(data, node$con) : error writing to connection. If I set the number of cores to 4, the parallel processing is OK (no errors). I am getting this error when I am using the package's function ataStartCluster().

I found a workaround on this problem if I set cl <- parallel::makeCluster(8, type = "PSOCK") instead of the ataStartCluster(). I also compared the execution times when using the parallel::makeCluster() vs no parallel, to see if the parallel computation works as intended, and indeed it does.

When using cl <- parallel::makeCluster(8, type = "PSOCK") the execution time is 1.5 minutes whereas when NOT using the parallel code the execution time is 3.2 minutes. The thing is, I don't have this issue on my other laptop which has 4gb of RAM and Windows 10 but with the same R version and packages as the ones I am using on this laptop (please see session info below).

The code I am using:

library(atakrig)
library(terra)
rpath <- system.file("extdata", package="atakrig")
aod3k <- rast(file.path(rpath, "MOD04_3K_A2017042.tif"))
aod10 <- rast(file.path(rpath, "MOD04_L2_A2017042.tif"))
aod3k.d <- discretizeRaster(aod3k, 1500)
aod10.d <- discretizeRaster(aod10, 1500)
grid.pred <- discretizeRaster(aod3k, 1500, type = "all")
aod3k.d$areaValues$value <- log(aod3k.d$areaValues$value)
aod10.d$areaValues$value <- log(aod10.d$areaValues$value)
## area-to-area Kriging ---# point-scale variogram from combined AOD-3k and AOD-10
aod.combine <- rbindDiscreteArea(x = aod3k.d, y = aod10.d)
vgm.ok_combine <- deconvPointVgm(aod.combine, model="Exp", ngroup=12, rd=0.75)
# point-scale cross-variogram
aod.list <- list(aod3k=aod3k.d, aod10=aod10.d)
aod.list <- list(aod3k=aod3k.d, aod10=aod10.d)
vgm.ck <- deconvPointVgmForCoKriging(aod.list, model="Exp", ngroup=12, rd=0.75,fixed.range = 9e4)
# prediction
cl <- parallel::makeCluster(8, type = "PSOCK")
pred.ataok <- ataKriging(aod10.d, grid.pred, vgm.ck$aod10, showProgress = TRUE, nopar = FALSE)
parallel::stopCluster(cl)

Session info:

R version 4.4.3 (2025-02-28 ucrt)
Platform: x86_64-w64-mingw32/x64
Running under: Windows 11 x64 (build 26100)

Matrix products: default


locale:
[1] LC_COLLATE=English_United States.utf8  LC_CTYPE=English_United States.utf8    LC_MONETARY=English_United States.utf8
[4] LC_NUMERIC=C                           LC_TIME=English_United States.utf8    


attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] terra_1.8-42    atakrig_0.9.8.1

loaded via a namespace (and not attached):
 [1] DBI_1.2.3          KernSmooth_2.23-26 sf_1.0-20          doSNOW_1.0.20      zoo_1.8-13         spacetime_1.3-3    xts_0.14.1        
 [8] e1071_1.7-16       snow_0.4-4         sp_2.2-0           gstat_2.1-3        grid_4.4.3         classInt_0.4-11    foreach_1.5.2     
[15] FNN_1.1.4.1        intervals_0.15.5   compiler_4.4.3     codetools_0.2-20   Rcpp_1.0.14        rstudioapi_0.17.1  lattice_0.22-7    
[22] class_7.3-23       parallel_4.4.3     magrittr_2.0.3     tools_4.4.3        proxy_0.4-27       iterators_1.0.14   units_0.8-7

EDIT

By calling the package foreach (i.e., library(foreach)) I made the code run succefully. I don't know why and how that helped.

Improve this question

1 Answer 1

Reset to default
1

I'm not sure how ataKriging might use the clusters using just cl <- parallel::makeCluster(8, type = "PSOCK") without registerDoSNOW the cluster and set the options that ataKriging wants. Moreover you show nopar=TRUE to be set. Maybe try parallel::makePSOCKcluster which is "an enhanced version of snow::makeSOCKcluster".

> ncpu <- parallel::detectCores() - 1L
> 
> cl <- parallel::makePSOCKcluster(ncpu)  ## make PSOCK cluster
> doSNOW::registerDoSNOW(cl)  ## register cluster
> options(ataKrigCluster=cl)  ## set options for atakrig
> 
> system.time(pred.ataok <- ataKriging(aod10.d, grid.pred, vgm.ck$aod10, showProgress=TRUE))
  |==================================================| 100%
   user  system elapsed 
  2.069   0.186  10.821 
> 
> parallel::stopCluster(cl)  ## stop cluster
> 
> head(pred.ataok)
  areaId    centx   centy     pred       var
1      1 857313.2 4487148 3.246404 0.4251597
2      2 860313.2 4487148 3.254721 0.4046833
3      3 863313.2 4487148 3.238521 0.3841170
4      4 866313.2 4487148 3.221157 0.3632869
5      5 869313.2 4487148 3.202737 0.3421746
6      6 872313.2 4487148 3.183429 0.3208301

For me, ataStartCluster also works, I'm on Linux, though.

> ataStartCluster(spec=ncpu)
socket cluster with 15 nodes on host ‘localhost’
> system.time(pred.ataok <- ataKriging(aod10.d, grid.pred, vgm.ck$aod10, showProgress=TRUE))
  |==================================================| 100%
   user  system elapsed 
  2.096   0.196  10.845 
> ataStopCluster()

Might be a problem with an outdated OS on one of your machines. (You might install Linux there, which is free ;)

Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

Regarding Linux, I am planning to install them on my Windows 10 laptop very soon. As for the atakrig I also made a mistake on my post (I corrected it now), I had nopar = TRUE when I was calling the parallel::makeCluster(), where I had to set nopar = FALSE. I think atakrig is calling snow internally because on the package's example they don't call the (snow) package, they just set ataStartCluster(2), but maybe I am wrong.
thank you for the code, it worked. But as I mentioned on my post all I had to do is to call the library foreach and not the parallel and doParallel. I really don't know why that worked (I am talking about my Windows 11 machine).
Just type ataStartCluster (w/o parentehses) and you'll see what it does. I can't imagine how loading foreach could have anything to do with it.
I just re-checked my claim and I am right. I terminated R, I didn't call foreach and I called parallel and doParallel and I got the error Error in serialize(data, node$con) : error writing to connection. Then I terminated R again only this time I called foreach and not the other two packages and the ataStartCluster(8) worked. I have very limited (to no) knowledge on how parallel computing works and how internally atakrig works or if there are conflicts between the packages, so all I can say is that my claim (on Windows 11) is correct. Sorry about that.
This doesn't explain how this is is possible—strange. Did you load library(atakrig)? Just noticed it loads parallel, doSNOW, snow,` and foreach as well as terra. If you think this is a bug, you might want to contact maintainer at github to they can fix it.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.