memory error with itertools when parallelizing for loop

Question

I'm trying to parallelize a for loop in python. I get a memory error in the permutations phase of the parameters.

I can see why this might fail, but what is the other way to do the same thing.

import itertools
import multiprocessing
LOAD_GEN_KEYS = range(138259)
ES_DATA_K = range(9606834)
paramlist      = list(itertools.product(LOAD_GEN_KEYS, ES_DATA_K))
pool           = multiprocessing.Pool()
VALID_TS       = pool.map(curate_results, paramlist)


def curate_results(params):
    LG = params[0]
    ES = params[1]
    ES_S = str(int(ES)/1000)
    if ES_S == LG:
        return [LG, ES]
    else:
        return []

Any help will be appreciated.

Alfe · Accepted Answer · 2018-01-05 14:48:54Z

2

You are creating a list of 138259 * 9606834 items. That is too much for your memory.

I propose to go with the generator which itertools.product() actually is (i. e. the elements aren't stored in memory but created on the fly during iterating):

paramlist      = itertools.product(LOAD_GEN_KEYS, ES_DATA_K)

Since the pool.map(curate_results, paramlist) accepts an iterable, a generator should do as fine as the list.

answered Jan 5, 2018 at 14:48

Alfe

60.2k21 gold badges117 silver badges172 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

tandem Over a year ago

looks like even doing that doesn't help it. should I do it in chunks instead?

tandem Over a year ago

` VALID_TS = pool.map(curate_results, paramlist) File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map return self.map_async(func, iterable, chunksize).get() File "/usr/lib/python2.7/multiprocessing/pool.py", line 304, in map_async iterable = list(iterable) MemoryError` Here is the error

Alfe Over a year ago

You are aware that you are mapping 1.3 trillion combinations, right? Where do you expect the results to be stored?

tandem Over a year ago

that's true again. don't think my dram is large enough to store all this..

tandem Over a year ago

I think that's the valid question. I need to ponder on this for a while.

|

Collectives™ on Stack Overflow

memory error with itertools when parallelizing for loop

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related