Clear memory in loop with R

Question

I'm running low with memory and speed as the loop proceeds. If I would place gc() in loop right after write.csv(), would that be correct and of any help?

Loop I have got:

for(i in seq_along(x) {
 ....
 ....
 write.csv(x, file=paste("C:/....",i,".csv",sep=""))
}

I suppose you mean paste("C:/....", i, ".csv", sep="")) and also x[[i]] instead of x. Consider providing some more info of what you are doing in the loop, are you creating temporary objects and then remove them? — alko989
– alko989, Commented May 23, 2014 at 19:18
No, actually I'm doing some scraping, that requires me to do it in steps. So the data is stored in steps. All works, just wondering if I could release some memory while in loop, after each i stored data. — Maximilian
– Maximilian, Commented May 23, 2014 at 19:24
I'm not sure why you couldn't just try this yourself and see. My guess, though, is that it won't help much, if at all. — joran
– joran, Commented May 23, 2014 at 20:04

alko989 · Accepted Answer · 2014-05-23 21:02:20Z

The garbage collector is called automatically when needed. Using gc() calls the garbage collector. I think, it makes only sense to use it if you remove objects in the loop. Then calling the garbage collector could help. Quoting from ?gc:

"[...] it can be useful to call ‘gc’ after a large object has been removed, as this may prompt R to return memory to the operating system."

Calling gc() can be time consuming. I did a little test to check that:

library(microbenchmark)
library(ggplot2)
lst <- rep(list(rnorm(10000)), 30)

res <- microbenchmark(
  for(i in seq_along(lst)) {
    write.csv(lst[[i]], file="delme.csv")
    gc()
  }, 
  for(i in seq(ll)) {
    write.csv(lst[[i]], file="delme.csv")
  })

levels(res$expr) <- c("with gc()","without gc()")
autoplot(res)

enter image description here

So it seems that calling gc() everytime is probably not a good idea. Of course it depends a lot on what you are doing in the loop.

Just a hunch: Garbage collection problems are not slowing your code down. You can probably optimize other parts of your code, e.g. using an *ply function instead of for loop can sometimes help.

Hope it helps,

alex

Collectives™ on Stack Overflow

Clear memory in loop with R

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related