Python threads and queue example

Question

I'm new to python (I come from PHP), I've been reading tutorials and trying things for a couple of days but I can't understand this queue example (http://docs.python.org/2/library/queue.html)

def worker():
    while True:
        item = q.get()
        do_work(item)
        q.task_done()

q = Queue()
for i in range(num_worker_threads):
     t = Thread(target=worker)
     t.daemon = True
     t.start()

for item in source():
    q.put(item)

q.join()       # block until all tasks are done

The thing I don't understand is how the worker thread completes and exists. I've read q.get() blocks until an item is available, so if all items are processed and none is left in the queue why q.get() doesn't block forever?

Pavel Anossov · Accepted Answer · 2013-03-02 11:47:02Z

9

Threads do not exit normally in this code (they are indeed blocked when the queue is empty). The program doesn't wait for them because they're daemon threads.

The program doesn't exit immediately and doesn't block forever because of q.join and q.task_done calls.

The count of unfinished tasks goes up whenever an item is added to the queue. The count goes down whenever a consumer thread calls task_done() to indicate that the item was retrieved and all work on it is complete. When the count of unfinished tasks drops to zero, join() unblocks, and the program exists without waiting for daemon threads.

edited Mar 2, 2013 at 11:47

answered Mar 2, 2013 at 11:34

Pavel Anossov

63.3k16 gold badges156 silver badges125 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Anton Danylchenko · Accepted Answer · 2014-01-31 12:06:44Z

-3

I have had the same problem. When all threads completed, I saw "sleeping threads" in process list (use top -H -p <pid> where <pid> is process id from ps aux | grep python with your script).

I solved this problem by replacing "infinite loop" while True to while not q.empty():.

It fixed the problem with "sleeping threads".

def worker():
    while not q.empty():
        item = q.get()
        do_work(item)
        q.task_done()

q = Queue()
for i in range(num_worker_threads):
     t = Thread(target=worker)
     t.daemon = True
     t.start()

for item in source():
    q.put(item)

q.join()       # block until all tasks are done

answered Jan 31, 2014 at 12:06

Anton Danylchenko

2,3671 gold badge26 silver badges25 bronze badges

1 Comment

me_and Over a year ago

That solution is rather risky. If the worker threads are quick, they'll test q.empty() before any items have been added to the queue, and then exit before they've done anything. Equally, if there's a single item left in the queue and two threads test q.empty() simultaneously, they'll both continue but one will get an item off the queue while the other will block at q.get().

Collectives™ on Stack Overflow

Python threads and queue example

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related