Python3 multiprocessing: Memory Allocation Error

Question

I know that this question has been asked a lot of times, but the answers are not applicable. This is answer one of a parallelized loop using multiprocessing on StackoverFlow:

import multiprocessing as mp

def processInput(i):
    return i * i

if __name__ == '__main__':
    inputs = range(1000000)
    pool = mp.Pool(processes=4)
    results = pool.map(processInput, inputs)
    print(results)

This code works fine. But if I increase the range to 1000000000, my 16GB of Ram are getting filled completely and I get [Errno 12] Cannot allocate memory. It seems as if the map function starts as many processes as possible. How do I limit the number of parallel processes?

Roy · Accepted Answer · 2017-10-23 07:46:59Z

1

The pool.map function starts 4 processes as you instructed it (in the line processes=4 you instruct the pool on how many processes it can use to perform your logic).

There is however a different issue underlying this implementation. The pool.map function will return a list of objects, in this case its numbers. Numbers do not act like int-s in ANSI-C they have overhead and will not overflow (e.g. turn to -2^31 whenever reaching 2^31+1 on 32-bit). Also python lists are not array and do incur an overhead.

To be more specific, on python 3.6, running the following code will reveal some overhead:

>>>import sys
>>>t = [1,2,3,4]
>>>sys.getsizeof(t)
96
>>>t = [x for x in range(1000)]
>>>sys.getsizeof(t)
9024

So this means 24 bytes per number on small lists and ~9 bytes on large lists. So for a list the size of 10^9 we get about 8.5GB

EDIT: 1. As tfb mentioned, this is not even the size of the underlying Number objects, just pointers and list overhead, meaning there is much more memory overhead I did not account for in the original answer.

Default python installation on windows is 32-bit (you can get 64-bit installation but you need to check the section of all available downloads in the python website), So I assumed you are using the 32-bit installation.

edited Oct 23, 2017 at 7:46

answered Oct 22, 2017 at 10:44

Roy

7978 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user5920214 Over a year ago

I'm pretty sure that sys.getsizeof just tells you the size of the array object, not its contents. For instance 1.0*sys.getsizeof([[i for i in xrange(10)] for j in xrange(10000)])/10000 is about 8.7 on my implementation. So clearly it's not counting the objects in the array, just pointers to them. And sys.getsizeof(1) is 24. (Python doesn't seem to do anything clever with tagging so fixnums can be immediate as far as I can see.)

user5920214 · Accepted Answer · 2017-09-25 14:44:14Z

0

range(1000000000) creates a list of 10^9 ints. This is around 8GB (8 bytes per int on a 64-bit system). You are then trying to process this to create another list of 10^9 ints. A really really smart implementation might be able to do this on a 16GB machine, but its basically a lost cause.

In Python 2 you could try using xrange which might or might not help. I am not sure what the Python 3 equivalent is.

answered Sep 25, 2017 at 14:44

user5920214

2 Comments

Roy Over a year ago

python 3 has deprecated the range implementation for the xrange generator, range() in python 3 is equal to using xrange() in python 2.

user5920214 Over a year ago

@Fanchi Thanks. I'm probably going to delete this answer because I think yours is better.

Collectives™ on Stack Overflow

Python3 multiprocessing: Memory Allocation Error

2 Answers 2

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related