Multiprocessing in python - UnboundLocalError: local variable 'data' referenced before assignment

Question

I am trying to access an API which to return a set of products. Since the execution is slow I was hoping could use multiprocessing to make it faster. The API works perfectly when accessed using a simple for loop.

Here is my code:

from multiprocessing import Pool
from urllib2 import Request, urlopen, URLError
import json

def f(a):
    request = Request('API'+ str(a)) 
    try:
        response = urlopen(request)
        data = response.read()
    except URLError, e:
        print 'URL ERROR:', e
    s=json.loads(data)
    #count += len(s['Results'])
    #print count
    products=[]
    for i in range(len(s['Results'])):
        if (s['Results'][i]['IsSyndicated']==False):        
            try:
                products.append(int(s['Results'][i]['ProductId']))
            except ValueError as e:
                products.append(s['Results'][i]['ProductId'])
    return products

list=[0,100,200]

if __name__ == '__main__':
    p = Pool(4)
    result=p.map(f, list)
    print result

Here is the error message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\z080302\Desktop\WinPython-32bit-2.7.6.3\python-2.7.6\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 540, in runfile
    execfile(filename, namespace)
  File "C:/Users/z080302/Desktop/Python_Projects/mp_test.py", line 36, in <module>
    result=p.map(f, list)
  File "C:\Users\z080302\Desktop\WinPython-32bit-2.7.6.3\python-2.7.6\lib\multiprocessing\pool.py", line 250, in map
    return self.map_async(func, iterable, chunksize).get()
  File "C:\Users\z080302\Desktop\WinPython-32bit-2.7.6.3\python-2.7.6\lib\multiprocessing\pool.py", line 554, in get
    raise self._value
UnboundLocalError: local variable 'data' referenced before assignment

I was thinking even with multiprocessing the function will still be executed sequentially. So why am I getting UnboundLocalError?

If an error is thrown in the try block, data will never be assigned. — Holloway
– Holloway, Commented Feb 27, 2015 at 15:19
Most likely because the urlopen(request) command is throwing an exception, so data is never bound. — dano
– dano, Commented Feb 27, 2015 at 15:19
Check your indentation. Also, when pasting python code - it matters a lot. — MustSeeMelons
– MustSeeMelons, Commented Feb 27, 2015 at 15:21

larsks · Accepted Answer · 2015-02-27 15:20:06Z

2

In this code:

try:
    response = urlopen(request)
    data = response.read()
except URLError, e:
    print 'URL ERROR:', e

If urlopen throws a URLError exception, the following line (data = response.read() is never executed. So when you come to:

s=json.loads(data)

The variable data has never been assigned. You probably want to abort processing in the event of a URLError, since that suggests you will not have any JSON data.

answered Feb 27, 2015 at 15:20

larsks

318k50 gold badges474 silver badges482 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

jbasko · Accepted Answer · 2017-04-27 17:21:01Z

2

The accepted answer is about the actual problem, but I thought I'll add my experience for others who come here because of mystic errors raised by multiprocessings ApplyResult.get with raise self._value. If you are getting TypeError, ValueError or basically any other error which in your case has nothing to do with multiprocessing then it's because that error is raised not by multiprocessing really, but by your code that you are running in the process that you are attempting to manage (or the thread if you happen to be using multiprocessing.pool.ThreadPool which I was).

answered Apr 27, 2017 at 17:21

jbasko

7,3603 gold badges42 silver badges52 bronze badges

Collectives™ on Stack Overflow

Multiprocessing in python - UnboundLocalError: local variable 'data' referenced before assignment

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related