The thread running under multiprocessing.Process does not update its instance attributes [closed]

Question

Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.

This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.

Closed 10 months ago.

Improve this question

I want to run several class instances in parallel and update the instance attributes using Redis. On each class instance, I run a thread in the background to listen to redis for changes. When I run set value 9 in redis, the thread detects the change, but the self.value attribute is not updated.

import time
import threading
import redis


class Foo:
    def __init__(self, name: str):
        self.name = name

        self.redis_client = redis.Redis(host="localhost", port=6379)
        self.value = 5

        self.update_thread = threading.Thread(target=self.update, daemon=True)
        self.update_thread.start()

    def update(self):
        while True:
            new_value = int(self.redis_client.get("value"))
            if new_value != self.value:
                print("new value detected {}".format(new_value))
                self.value = new_value
            time.sleep(2)

    def run(self):
        while True:
            print("{}: Value: {}".format(self.name, self.value))
            time.sleep(2)


if __name__ == "__main__":
    import multiprocessing

    foo1 = Foo(name="foo1")
    foo2 = Foo(name="foo2")

    process1 = multiprocessing.Process(target=foo1.run)
    process2 = multiprocessing.Process(target=foo2.run)

    process1.start()
    process2.start()

console output:

foo1: Value: 5
foo2: Value: 5
new value detected 9
new value detected 9
foo1: Value: 5
foo2: Value: 5

When I research the subject, I come across the information that "each process has its own memory space". But since I do not share data between processes in this case, I cannot understand why the data in the object instances cannot be preserved.

Multithreading is not the same as multiprocessing. Processes have distinct address spaces, so a variable changing in one process won't change the same variable in a different process. — Ulrich Eckhardt
– Ulrich Eckhardt, Commented Feb 12 at 14:32

juanpa.arrivillaga · Accepted Answer · 2025-02-12 16:38:42Z

You are sharing data between processes, the main process and the subprocesses you created.

You create a thread in the main process when you instantiate foo, e.g.:

foo1 = Foo(name="foo1")

You then create a subprocess:

process1 = multiprocessing.Process(target=foo1.run)
...
process1.start()

What this does is essentially pickle foo1, send that object over the wire to the subprocess that was created, and in that subprocess the instance is unpickled and foo1.run is called. Now, in your main process, the thread that is running foo1.update is updating the instance in the main process, but your instance in the subprocess is not being updated, so it always reports the original value.

So try adding something like:

if __name__ == "__main__":

    import multiprocessing

    foo1 = Foo(name="foo1")
    foo2 = Foo(name="foo2")

    process1 = multiprocessing.Process(target=foo1.run)
    process2 = multiprocessing.Process(target=foo2.run)

    process1.start()
    process2.start()

    while True:
        print(f"In the main process, {foo1.value=}")
        time.sleep(10)

Perhaps, it is important to understand what a method is.

>>> class Foo:
...     def bar(self):
...         print("hi from", self)
...

When accessed on the class, a method is just a function!:

>>> Foo.bar
<function Foo.bar at 0x10120cd30>
>>> Foo.bar()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Foo.bar() missing 1 required positional argument: 'self'

But if you access it on an instance (through the magic of the descriptor protocol, read this whole thing but pay attention to this section) it returns a bound method object. That is basically a special callable object that passes the instance to the first positional argument (the name self is just a convention, you could call it banana if you want).

>>> foo = Foo()
>>> bound_method = foo.bar
>>> bound_method
<bound method Foo.bar of <__main__.Foo object at 0x10121beb0>>
>>> bound_method.__func__
<function Foo.bar at 0x10120cd30>
>>> bound_method.__self__
<__main__.Foo object at 0x10121beb0>
>>> bound_method()
hi from <__main__.Foo object at 0x10121beb0>

So the argument so multiprocessing.Process(target=...) will be pickled, and this argument is a bound-method object, which is basically a reference to the function (the method!) and the instance.

Collectives™ on Stack Overflow

The thread running under multiprocessing.Process does not update its instance attributes [closed]

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related