Python multiprocessing making same object instance for every process

Question

I have written a simple example to illustrate what exactly I'm banging my head onto. Probably there is some very simple explanaition that I just miss.

import time
import multiprocessing as mp
import os


class SomeOtherClass:
    def __init__(self):
        self.a = 'b'


class SomeProcessor(mp.Process):
    def __init__(self, queue):
        super().__init__()
        self.queue = queue

    def run(self):
        soc = SomeOtherClass()
        print("PID: ", os.getpid())
        print(soc)

if __name__ == "__main__":
    queue = mp.Queue()

    for n in range(10):
        queue.put(n)

    processes = []

    for proc in range(mp.cpu_count()):
        p = SomeProcessor(queue)
        p.start()
        processes.append(p)

    for p in processes:
        p.join()

Result is:

PID: 11853
<__main__.SomeOtherClass object at 0x7fa637d3f588>
PID: 11854
<__main__.SomeOtherClass object at 0x7fa637d3f588>
PID: 11855
<__main__.SomeOtherClass object at 0x7fa637d3f588>
PID: 11856
<__main__.SomeOtherClass object at 0x7fa637d3f588>

Object address is the same for all, regardless every initialization happened in a new process. Can anyone point out what's the problem. Thanks.

Also I wonder about this behaviour, when I first initialize the same object in the main process then cache some values on it and then initialize the same object on every process. Then the processes inherit the main process object.

import time
import multiprocessing as mp
import os
import random

class SomeOtherClass:

    c = {}

    def get(self, a):
        if a in self.c:
            print('Retrieved cached value ...')
            return self.c[a]

        b = random.randint(1,999)

        self.c[a] = b

        return b


class SomeProcessor(mp.Process):
    def __init__(self, queue):
        super().__init__()
        self.queue = queue

    def run(self):
        pid = os.getpid()
        soc = SomeOtherClass()
        val = soc.get('new')
        print("Value from process {0} is {1}".format(pid, val))

if __name__ == "__main__":
    queue = mp.Queue()

    for n in range(10):
        queue.put(n)

    pid = os.getpid()
    soc = SomeOtherClass()
    val = soc.get('new')
    print("Value from main process {0} is {1}".format(pid, val))

    processes = []

    for proc in range(mp.cpu_count()):
        p = SomeProcessor(queue)
        p.start()
        processes.append(p)

    for p in processes:
        p.join()

Output here is :

Value from main process 13052 is 676
Retrieved cached value ...
Value from process 13054 is 676
Retrieved cached value ...
Value from process 13056 is 676
Retrieved cached value ...
Value from process 13057 is 676
Retrieved cached value ...
Value from process 13055 is 676

Sorry to not mention that. Exact version of Python outputing this result is 3.6.9 — Mario Kirov
– Mario Kirov, Commented Sep 7, 2021 at 11:26
There is no problem here. The instances are in different processes and don't share state. — AKX
– AKX, Commented Sep 7, 2021 at 11:33
@AKX I can prove you wrong. Exactly that's why I'm asking of this behaviour. — Mario Kirov
– Mario Kirov, Commented Sep 7, 2021 at 11:34
@MarioKirov You declare SomeOtherClass.c as a class-level variable. It will be shared between all SomeOtherClassinstances in the same process too. (If you want it to be instance-level, you'll need to do self.c = {} in __init__.) By virtue of forking, the same value will be in the child processes too. — AKX
– AKX, Commented Sep 7, 2021 at 13:16

AKX · Accepted Answer · 2021-09-07 13:45:31Z

8

To expand on the comments and discussion:

On Linux, multiprocessing defaults to the fork start method. Forking a process means child processes will share a copy-on-write version of the parent process's data. This is why the globally created objects have the same address in the subprocesses.
- On macOS and Windows, the default start method is spawn – no objects are shared in that case.
The subprocesses will have their unique copies of the objects as soon as they write to them (and internally in CPython, in fact, when they even read them, due to the reference counter being in the object header).
A variable defined as
```
class SomeClass:
    container = {}
```
is class-level, not instance-level and will be shared between all instances of SomeClass. That is,
```
a = SomeClass()
b = SomeClass()
print(a is b)  # False
print(a.container is b.container is SomeClass.container)  # True
a.container["x"] = True
print("x" in b.container)  # True
print("x" in SomeClass.container)  # True
```
By virtue of the class's state being forked into the subprocess, the shared container also seems shared. However, writing into the container in a subprocess will not appear in the parent or sibling processes. Only certain special multiprocessing types (and certain lower-level primitives) can span process boundaries.
To correctly separate that container between instances and processes, it will need to be instance-level:
```
class SomeClass:
    def __init__(self):
        self.container = {}
```
(However, of course, if a SomeClass is globally instantiated, and a process is forked, its state at the time of the fork will be available in subprocesses.)

edited Sep 7, 2021 at 13:45

answered Sep 7, 2021 at 13:38

AKX

171k16 gold badges147 silver badges229 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Mario Kirov Over a year ago

Very well explained. Thank you one more time for the deep and simple explanations.

Booboo Over a year ago

I am not sure it fully explains what the OP sees. I agree at the time of the fork all objects in the forked processes will have the same address as the main process. But in the OP's case each process is now continuing to run and each process is creating new objects. It is these new objects in each address space that have the same address. Now you can argue that each forked process is proceeding identically to one another and that is why these objects will have the same address. But you did not argue that. If memory allocation used a random number generator, you would not see this.

Shinra tensei · Accepted Answer · 2021-09-08 10:47:06Z

1

tldr: They're actually not the same instance, so don't worry about that.

Well that's interesting. Their memory reference is exactly the same, but the instances are definitely different. If we modify the code like this:

import time
import multiprocessing as mp
import os


class SomeOtherClass:
    def __init__(self, num):
        self.a = num  # <-- Let's identify the instance with the pid
    
    def __str__(self):
        return f"I'm number {self.a}"  # <-- Better representation of the instance


class SomeProcessor(mp.Process):
    def __init__(self, queue):
        super().__init__()
        self.queue = queue

    def run(self):
        soc = SomeOtherClass(os.getpid())  <-- Use the PID to instantiate different objects
        print("PID: ", os.getpid())
        print(soc)
        time.sleep(1)
        print(soc)  # <-- Give it a second and print again

if __name__ == "__main__":
    queue = mp.Queue()

    for n in range(10):
        queue.put(n)

    processes = []

    for proc in range(mp.cpu_count()):
        p = SomeProcessor(queue)
        p.start()
        processes.append(p)

    for p in processes:
        p.join()

We can see that the instances are definitely different, and they aren't being modified, because after the time.sleep() they still have their attributes unchanged:

PID:  668424
I'm number 668424
PID:  668425
I'm number 668425
PID:  668426
I'm number 668426
...
I'm number 668435
I'm number 668424
I'm number 668426
...

Yet, if we remove the __str__ function, I still see the same memory reference:

<__main__.SomeOtherClass object at 0x7f3e08d83bb0>
PID:  669008
<__main__.SomeOtherClass object at 0x7f3e08d83bb0>
PID:  669009
<__main__.SomeOtherClass object at 0x7f3e08d83bb0>
PID:  669010
...
<__main__.SomeOtherClass object at 0x7f3e08d83bb0>
<__main__.SomeOtherClass object at 0x7f3e08d83bb0>
<__main__.SomeOtherClass object at 0x7f3e08d83bb0>
...

To be honest, I don't really know the reason why this happens, so other people could help you more. As the user Booboo has said, you're seeing this because of the fact that Linux uses fork to start a new process. I did run this in a Linux machine too. If Windows had been used, the memory reference would be different.

edited Sep 8, 2021 at 10:47

answered Sep 7, 2021 at 11:36

Shinra tensei

1,3139 silver badges22 bronze badges

4 Comments

AKX Over a year ago

The processes are parallel alright. The GIL does not span processes, and during a fork there's no pausing or pickling involved.

Shinra tensei Over a year ago

@AKX no, not during the fork, but during the change of process. afaik, multiprocessing uses pickle in order to save the states of the objects in the process, so I thought it was possible that the instances that were being unpickled ended up in the same memory address as the ones in the other processes. However, it's true that multiprocessing side-steps the GIL

AKX Over a year ago

Forking is "the change of process". Multiprocessing only serializes (e.g. pickles) objects when you're explicitly sending them across processes, e.g. with queues.

Shinra tensei Over a year ago

All right, I understand what you're saying and it's a good explanation. There's only one thing we both might be misunderstanding: 'Forking is "the change of process"'. I might not be expressing myself correctly, when I say "the change of process" I'm not talking about the creation of a new process, I already know that's what forking is, I'm rather talking about the context switching. There's no forking involved there.

balderman · Accepted Answer · 2021-09-07 11:39:51Z

0

Look at the modified code that shows that every SomeOtherClass is different.

import time
import multiprocessing as mp
import os


class SomeOtherClass:

  def __new__(cls, *args, **kwargs):
        print('-- inside __new__ --')
        return super(SomeOtherClass, cls).__new__(cls, *args, **kwargs)


    def __init__(self):
        self.a = os.getpid()
    def __str__(self):
        return f'{self.a}'


class SomeProcessor(mp.Process):
    def __init__(self, queue):
        super().__init__()
        self.queue = queue

    def run(self):
        soc = SomeOtherClass()
        print("PID: ", os.getpid())
        print(soc)

if __name__ == "__main__":
    queue = mp.Queue()

    for n in range(10):
        queue.put(n)

    processes = []

    for proc in range(mp.cpu_count()):
        p = SomeProcessor(queue)
        p.start()
        processes.append(p)

    for p in processes:
        p.join()

output

 -- inside __new__ --
PID:  25054
25054
-- inside __new__ --
PID:  25055
25055
-- inside __new__ --
PID:  25056
25056
-- inside __new__ --
PID:  25057
25057
-- inside __new__ --
PID:  25058
25058
-- inside __new__ --
PID:  25059
25059
-- inside __new__ --
PID:  25060
25060
-- inside __new__ --
PID:  25061
25061

edited Sep 7, 2021 at 11:39

answered Sep 7, 2021 at 11:31

balderman

24k8 gold badges39 silver badges60 bronze badges

8 Comments

Mario Kirov Over a year ago

True and not true. They are on a different process, indeed. But what i want to accomplish is having a new instance not the same instance on a different process.

balderman Over a year ago

` same instance on a different process` - this is not the situation since we have a clear evidence the __init__ was called N times. Isnt it?

Booboo Over a year ago

@MarioKirov They are different instances by dint of being in different address spaces. They just happened to have the same address. I conjectured that you are running under Linux. If you were running under Windows you would probably not see this. Just guessing . When I run this under Linux I get the same addresses but not under Windows. Linux creates processes using fork vs. spawn for Windows.

balderman Over a year ago

I have added __new__ implementation in order to convince you that it is NOT the same instance. Look at the code.

Mario Kirov Over a year ago

@balderman Ok in this case you are right, they are different, my bad, just because I cut down the full example. Now added it in the question below the first one. Thanks

|

Collectives™ on Stack Overflow

Python multiprocessing making same object instance for every process

3 Answers 3

2 Comments

4 Comments

8 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

4 Comments

8 Comments

Your Answer

Sign up or log in

Post as a guest

Related