python "local variable referenced before assignment" with hundreds of threads

Question

I am having a problem with a piece of code that is executed inside a thread in python. Everything works fine until I start using more than 100 or 150 threads, then I get the following error in several threads:

    resp.read(1)
UnboundLocalError: local variable 'resp' referenced before assignment.

The code is the following:

    try:
        resp = self.opener.open(request)
        code = 200
    except urllib2.HTTPError as e:
        code = e.code
        #print e.reason,_url
        #sys.stdout.flush()
    except urllib2.URLError as e:
        resp = None
        code = None

    try:
        if code:
            # ttfb (time to first byte)
            resp.read(1)
            ttfb = time.time() - start
            # ttlb (time to last byte)
            resp.read()
            ttlb = time.time() - start
        else:
            ttfb = 0
            ttlb = 0
    except httplib.IncompleteRead:
        pass

As you can see if "resp" is not assigned due to an exception, it should raise the exception and "code" coundn't be assigned so it couldn't enter in "resp.read(1)".

Anybody has some clue on wht it is failing? I guess it is related to scopes but I don't know how to avoid this or how to implement it differently.

Thanks and regards.

dmeister · Accepted Answer · 2013-04-30 13:46:29Z

5

Basic python: If there is a HttpError during the open call, resp will not be set, but code will be set to e.code in the exception handler. Then code is tested and resp.read(1) is called.

This has nothing to do with threads directly, but maybe the high number of threads caused the HTTPError.

answered Apr 30, 2013 at 13:46

dmeister

35.9k19 gold badges76 silver badges98 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

dbranco Over a year ago

I think that is not correct. The HTTPError is raised when there is a response code diferent from 200, for example a 404. It works perfectly with 404 errors. Besides, it worked perfectly with thousands of requests with and without errors and and low concurrency. The connection errors wich could cause that resp is not assigned are managed by URLError afaik as they are not HTTP errors (protocol errors). Thanks fo your response.

glglgl Over a year ago

@user2336071 If you are sure about that, please tell me what you think is referred to as resp in the case of a HTTPError?

Edward Loper Over a year ago

@user2336071 I agree with @dmeister that this is almost certainly the cause. In particular, the only way that you can get an exception in your first try block is from the call to opener.open. And if you do get an exception there, then resp will not get any value. At any rate, it's easy enough to test -- try adding resp=None to the except HTTPError block, and see if that changes the error from UnboundLocalError to AttributeError (which you'd get when you try to access resp.read, where resp==None

dbranco Over a year ago

@glglgl In case of HTTPError, for example a 404 Not Found, resp would be the file object containing the body and yes, a 404 can have a body.

dbranco Over a year ago

@EdwardLoper I tried adding resp=Non inside the except HTTPError and now I get a NoneType error, so that seems to be the problem and fails with 404 HTTP Errors. But anyway I don't understand why is code=200 running when it is after an exception and mostly why with less threads the 404 HTTP Errors do not cause that behaviour.

|

Ahmet DAL · Accepted Answer · 2013-04-30 16:40:14Z

0

Defining and using resp variable are not is same code block. One of them in a try/except, the other is in another try/except block. Try to merge them:

Edited:

ttfb = 0
ttlb = 0
try:
        resp = self.opener.open(request)
        code = 200
        resp.read(1)
        ttfb = time.time() - start
        resp.read()
        ttlb = time.time() - start         
     except urllib2.HTTPError as e:
        code = e.code
        #print e.reason,_url
        #sys.stdout.flush()
    except urllib2.URLError as e:
        pass
    except httplib.IncompleteRead:
        pass

edited Apr 30, 2013 at 16:40

answered Apr 30, 2013 at 13:45

Ahmet DAL

4,74210 gold badges52 silver badges75 bronze badges

1 Comment

dbranco Over a year ago

But that is not a solution. I could also add another except for the UnboundLocalError in the second try. The problem is why the 'resp' variable is not assigned ?

Collectives™ on Stack Overflow

python "local variable referenced before assignment" with hundreds of threads

2 Answers 2

8 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

8 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related