How do you share a variable between two processes in Python?

Question

I have two processes, one adds jobs to a queue and the other takes them off the same queue and runs them. This should work as expected and I'm not sure why the worker never gets any jobs. Here's my code:

from multiprocessing import Process
from Queue import Queue
import time

q = Queue()

def queuer():
    while True:
        q.put("JOB")
        print "Adding JOB"
        time.sleep(1)

def worker():  
    while True:
        if not q.empty():
            item = q.get()
            print "Running", item
        else:
            print "No jobs"
            time.sleep(1)

a = Process(target=queuer)
a.start()

b = Process(target=worker)
b.start()

each process has its own copy of the queue. There is another stackoverflow thread that discusses this: stackoverflow.com/questions/11109776/… — wizard23
– wizard23, Commented Mar 12, 2014 at 23:46

Felix · Accepted Answer · 2014-03-12 23:52:38Z

9

Two things:

You need to pass the Queue as an argument to both processes.
You should use multiprocessing.Queue, not Queue.Queue (which are for Threads)

This code works for me:

from multiprocessing import Process, Queue
import time

def queuer(q):
    while True:
        q.put("JOB")
        print "Adding JOB"
        time.sleep(1)

def worker(q):  
    while True:
        if not q.empty():
            item = q.get()
            print "Running", item
        else:
            print "No jobs"
            time.sleep(1)



if __name__ == '__main__':
    q = Queue()
    a = Process(target=queuer, args=(q,))
    b = Process(target=worker, args=(q,))
    a.start()
    b.start()

edited Mar 12, 2014 at 23:52

answered Mar 12, 2014 at 23:43

Felix

2,1744 gold badges23 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

wizard23 · Accepted Answer · 2014-03-13 00:00:31Z

One possibility is to use the Queue object from the multiprocessing namespace. It's described here: http://docs.python.org/2/library/multiprocessing.html#sharing-state-between-processes

So I adapted your code. I only made 2 changes: - Use multiprocessing Queue - Avoid globals and pass the queue as a parameter to the worker and queuer (this is not needed but it's good practice to keep everything tidy)

# use the Queue from the multiprocessing namespace!
from multiprocessing import Process, Queue
import time

q = Queue()

def queuer(q):
    while True:
        q.put("JOB")
        print "Adding JOB"
        time.sleep(1)

def worker(q):  
    while True:
        if not q.empty():
            item = q.get()
            print "Running", item
        else:
            print "No jobs"
            time.sleep(1)

a = Process(target=queuer, args =(q,))
a.start()

b = Process(target=worker, args = (q,))
b.start()

Collectives™ on Stack Overflow

How do you share a variable between two processes in Python?

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related