Python - Using threads within multiprocessing

Question

Can someone please help me understand the constraints of using threading from within a python process.

I have attached a minimal working example of what I am trying to achieve. My use case requires that I bring up several processes and from within each process I have two threads that need to communicate. However even within the very simplified example below I seem to be running into deadlock / contention and it's not at all clear what is going wrong.

import multiprocessing
from threading import Thread
import logging
import time
import sys


def print_all_the_things(char, num):
    try:
        while True:
            sys.stdout.write(char + str(num))
    except Exception:
        logging.exception("Something went wrong")


class MyProcess(multiprocessing.Process):
    def __init__(self, num):
        super(MyProcess, self).__init__()
        self.num = num

    def run(self):
        self.thread1 = Thread(target=print_all_the_things, args=("a", self.num))
        self.thread2 = Thread(target=print_all_the_things, args=("b", self.num))

        self.thread1.start()
        self.thread2.start()

procs = {}
for a in range(2):
    procs[a] = MyProcess(a)
    procs[a].start()

time.sleep(5)

for a in range(2):
    procs[a].join()

The expected output is a mishmash of 'a', 'b', '1' and '2' on stdout. However the program very quickly deadlocks:

$python mwe.py
a0a0a0a0a0a0b0b0a0a0b0b0b0b0b0b0b0b0b0b0a0a0a0a0a0a0a0a1a1a2a2a2a2a2b2b2a2

I should point out that changing MyProcess to inherit from Thread results in a working example.

What am I doing wrong?

mguijarr · Accepted Answer · 2015-06-24 14:00:06Z

4

The 2 processes are started, they start their threads, then they should exit since there are no more instructions in run(). But the threads remain in a kind of zombie state, because the 'daemon' flag has not been set (see Python documentation about this), preventing the 2 processes to terminate properly.

Just make run() method not finishing just after the threads are started, for example you can wait on an exit condition:

class MyProcess(multiprocessing.Process):
    def __init__(self, num, exit_cond): ### new code
        super(MyProcess, self).__init__()
        self.num = num
        self.exit_cond = exit_cond ### new code

    def run(self):
        self.thread1 = Thread(target=print_all_the_things, args=("a", self.num))
        self.thread2 = Thread(target=print_all_the_things, args=("b", self.num))
        self.thread1.daemon=True ### new code
        self.thread2.daemon=True ### new code

        self.thread1.start()
        self.thread2.start()
        self.exit_cond.wait() ### new code

procs = {}
exit_cond = multiprocessing.Event() ### new code

for a in range(2):
    procs[a] = MyProcess(a, exit_cond)
    procs[a].start()

time.sleep(5)

exit_cond.set() ### new code
for a in range(2):
    procs[a].join()

answered Jun 24, 2015 at 14:00

mguijarr

8,0306 gold badges50 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

sterne Over a year ago

This code works perfectly. I had thought that the process would be kept alive while it waited for the threads to finish, this led me to believe that it was an issue of deadlock. Thanks for clearing up my misunderstanding!

Collectives™ on Stack Overflow

Python - Using threads within multiprocessing

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related