multiprocessing.Queue weird python behavior

Question

I stuck with understanding quite simple case. Please, can someone explain or show the direction to understand the following:

import multiprocessing as mp

if __name__ == '__main__':
    input_queue = mp.Queue()
    for i in range(5):
        input_queue.put([i]*5)
    print(input_queue.qsize())
    while not input_queue.empty():
        o = input_queue.get()
        print(o)

Output:

5
[0, 0, 0, 0, 0]
[1, 1, 1, 1, 1]
[2, 2, 2, 2, 2]
[3, 3, 3, 3, 3]
[4, 4, 4, 4, 4]

But:

import multiprocessing as mp

if __name__ == '__main__':
    input_queue = mp.Queue()
    for i in range(5):
        input_queue.put([i]*5)
    # print(input_queue.qsize())
    while not input_queue.empty():
        o = input_queue.get()
        print(o)

Output nothing

Update:

import multiprocessing as mp

if __name__ == '__main__':
    input_queue = mp.Queue()
    for i in range(5):
        input_queue.put([i]*5)
    for _ in range(5):
        o = input_queue.get()
        print(o)

Print expected output. So probably issue in .empty() method.

python --version
Python 3.6.9 :: Anaconda, Inc.

I'm getting the same output from the second case (minus the queue size) as expected. Maybe there's an issue with your print output target? — user11837869
– user11837869, Commented Jul 1, 2020 at 16:37
I don't think so, because in other cases is woks expectedly. I update my question with an example. — D.Shaulskii
– D.Shaulskii, Commented Jul 1, 2020 at 16:44

kcontr · Accepted Answer · 2020-07-01 16:54:58Z

1

You could be hitting this

After putting an object on an empty queue there may be an infinitesimal delay before the queue’s empty() method returns False

Can you try adding a sleep on the line before while not input_queue.empty():?

import multiprocessing as mp
import time

if __name__ == '__main__':
    input_queue = mp.Queue()
    for i in range(5):
        input_queue.put([i]*5)
    # print(input_queue.qsize())
    time.sleep(1)
    while not input_queue.empty():
        o = input_queue.get()
        print(o)

If the above works, then your print(input_queue.qsize()) call in the first example is what is giving the queue enough time to pickle the objects and start returning False on the empty() call.

answered Jul 1, 2020 at 16:54

kcontr

6336 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

kcontr Over a year ago

Happy to help! Can you also "accept" the answer? Thanks

Collectives™ on Stack Overflow

multiprocessing.Queue weird python behavior

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related