36

After a short debate with someone about exception handling in Python - sparked by the handling of a queue object - I thought I'd throw it out there...

METHOD 1:

import Queue

q = Queue.Queue()

try:
    task=q.get(False)
    #Opt 1: Handle task here and call q.task_done()
except Queue.Empty:
    #Handle empty queue here
    pass

#Opt2: Handle task here and call q.task_done()

METHOD 2:

import Queue

q = Queue.Queue()

if q.empty():
    #Handle empty queue here
else:
    task = q.get()
    #Handle task here
    q.task_done()

One argument is that Method 1 is wrong because the queue being empty is not an error, and therefore should not be handled using Queue.Empty exception. Additionally, it could make debugging more difficult when coded this way if you consider that the task handling part could potentially large.

The other argument is that either way is acceptable in Python and that handling the task outside of the try/except could aid debugging if task handling is large, although agreed that this might look uglier than using Method 2.

Opinions?

UPDATE: A little more info after answer 1 came through.... The debate was started after method 1 was using in some multithreaded code. In which case, the code will acquire the lock (from a threading.Lock object) and release it either once the task it returned or Queue.Empty is thrown

UPDATE 2: It was unknown to both of us that the the queue object was thread safe. Looks like try/except is the way to go!

Improve this question
1
  • 1
    I don't see how a Lock changes the answer, I also don't understand why it needs a Lock, the Queue is already thread-safe. Commented Jun 28, 2012 at 15:23

5 Answers 5

Reset to default
61

Method 2 is wrong because you are doing an operation in two steps when it could be done in one. In method 2, you check if the queue is empty, and then later (very soon, but still later), try to get the item. What if you have two threads pulling items from the queue? The get() could still fail with an empty queue. What if an item is added to the queue after you checked that it was empty? These are the sort of tiny windows of opportunity where bugs creep in to concurrent code.

Do it in one step, it's by far the better choice.

import Queue

q = Queue.Queue()

try:
    task = q.get(False)
except Queue.Empty:
    # Handle empty queue here
    pass
else:
    # Handle task here and call q.task_done()

Don't get hung up on "exceptions should be errors". Exceptions are simply another channel of communication, use them. Use the "else" clause here to narrow the scope of the exception clause.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

import Queue doesn't work.in Python 3. Can this be updated?
in python3 it should then be import queue and queue.Empty.
for python 3 do : import queue and q = queue.Queue() and except queue.Empty:
8

If this is multithreaded / multiprocessed code (as is a good reason for using queues anyway), then definitely method 1. Between the q.empty() call and the q.get() call, the Jack of Hearts could have stolen your tarts!

Improve this answer

2 Comments

The debate was started after method 1 was using in some multithreaded code. In which case, the code will acquire the lock (from a threading.Lock object) and release it either once the task it returned or Queue.Empty is thrown ......same opinion?
keep the errorhandling code as tight as possible - call get and check for error straight after that. no need to embed the rest of the code in the try...
6

One argument is that Method 1 is wrong because the queue being empty is not an error, and therefore should not be handled using Queue.Empty exception

An exception is not necessarily an "error", it's a general flow control mechanism, and is indeed used that way in a few cases (SysExit, StopIteration etc).

The good question here is: what will be the most common case - empty or non-empty queue. Unless you know for sure, you want to AskBeforeYouLeap, cause it's very probably way cheaper.

Improve this answer

Comments

3

Faced the same issue of not finding Queue module; I know this is about a year later, but future readers may find the following model example helpful:

from queue import Queue, Full, Empty
q = Queue(maxsize=3)    # finite queue, maxsize <= 0 -> infinite queue

# enqueue a couple of items
q.put('a')
q.put('b')

# trying to dequeue excessively
for _ in range(4):
    try:
        print(f'dequeued: {q.get(block=False)}')

    except Empty:
        print('empty queue!')

# trying to enqueue excessively
for ch in ['a', 'b', 'c', 'd']:
    try:
        print(f'enqueueing : {ch} ->', end=' ')
        q.put(ch, block=False)
        print('success :)')

    except Full:
        print('full queue :(')

and here is the sample output:

dequeued: a
dequeued: b
empty queue!
empty queue!
enqueueing : a -> success :)
enqueueing : b -> success :)
enqueueing : c -> success :)
enqueueing : d -> full queue :(

Please note, by default, get() and put() are both blocking methods (block=True) and will block the process until a new item is inserted into the queue (get), or an item is consumed to allow room for a new item (put). See here for more details.

Improve this answer

4 Comments

it isn't clear how this answer addresses the question about best practices for exception handling
@shortorian, I thought the best answer is already given to the question by @Ned Batchelder; this sample code was meant to help with the fact that Queue module should be replaced with queue for Python 3, which is asked by @vy32.
ah, ok. SO rules are pretty strict about sticking to one topic at a time; if you had enough reputation I'd say you should post a new question with a link back to this discussion and answer it yourself. I think you don't have enough reputation to answer your own questions yet, so it would have been better to add a comment under @vy32's suggesting they post a new question you could then answer
Yes, I first tried to comment directly below @vy32's question, but it didn't let me do so, required 50 reputation :(
0

In some code I am writing right now, I need to rapidly empty a queue and only process the last item, which in my case is the most recent event, and is all that I care about.

I essentially use your Method 1 to do this, my code being something like:

# pop data from queue until it is empty and so have the most recent data:
data = self.queue.get()    # first call to the queue blocks, so when get past this line, data should be valid
try:
    while True:
        data = self.queue.get_nowait()    # subsequent calls do NOT block; overwrite data with the last valid result; this call will raise an Empty Exception after have read all data from the queue
except Empty:
    pass

If I understand Python's queue API correctly, this is the only good way that I can empty the queue.

That said, I hate Python's queue API!

Sorry Pythonistas, its defective. There ought to be a nonblocking way that I can poll the queue for its first item and it returns None if it is empty instead of raising an Empty. Using an Exception here is a terrible choice not only because it requires an awkward try/except but also because it is awful for performance: stacktraces are expensive to generate. The code above is in a real critical section for me.

Contrast Python's queue with, say Java's Queue interface poll method; see also this SO question. Infinitely better API design.

Is there a 3rd party Python concurrent queue implementation that has a poll like method?

Improve this answer

1 Comment

"only process the last item" sounds like you need a stack instead of a queue. Your point on performance is valid, since I had similar issues before. But in most applications, I can avoid using non-blocking, by isolating the blocking part into another thread so rest the code can continue running. Sometimes, I just need to add another thread to make Python queue work. As a I said before, your example code may need stack or deque: docs.python.org/3/library/collections.html#collections.deque

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.