threading with python : local variables trouble

Question

I have some trouble using threading and scipy.stats.randint module. Indeed, when several threads are launched, a local array (bootIndexs in the code below) seems to be used for all launched thread.

This is the raised Error

> Exception in thread Thread-559:
Traceback (most recent call last):
...
  File "..\calculDomaine3.py", line 223, in bootThread
    result = bootstrap(nbB, distMod)
  File "...\calculDomaine3.py", line 207, in bootstrap
    bootIndexs = spstats.randint.rvs(0, nbTirages-1, size = nbTirages)
  File "C:\Python27\lib\site-packages\scipy\stats\distributions.py", line 5014, in rvs
    return super(rv_discrete, self).rvs(*args, **kwargs)
  File "C:\Python27\lib\site-packages\scipy\stats\distributions.py", line 582, in rvs
    vals = reshape(vals, size)
  File "C:\Python27\lib\site-packages\numpy\core\fromnumeric.py", line 171, in reshape
    return reshape(newshape, order=order)
ValueError: total size of new array must be unchanged

And this is my code :

import threading
import Queue
from scipy import stats as spstats

nbThreads = 4

def test(nbBoots, nbTirages,  modules ):

    def bootstrap(nbBootsThread, distribModules) :

         distribMax = []            

         for j in range(nbBootsThread): 
             bootIndexs = spstats.randint.rvs(0, nbTirages-1, size = nbTirages) 
             boot = [distribModules[i] for i in bootIndexs]

             distribMax.append(max(boot))

         return distribMax

    q = Queue.Queue()

    def bootThread (nbB, distMod):
        result = bootstrap(nbB, distMod )
        q.put(result, False)
        q.task_done()

    works = []

    for i in range(nbThreads) :     
        works.append(threading.Thread(target = bootThread, args = (nbBoots//nbThreads, modules[:],) ))


    for w in works:
        w.daemon = True
        w.start()

    q.join()

        distMaxResult = []

        for j in range(q.qsize()):
            distMaxResult += q.get()

        return distMaxResult

class classTest:
    def __init__(self):
        self.launch()

    def launch(self):
        print test(100, 1000, range(1000) )

Thanks for your answers.

I don't see anything in your code that would explain the behaviour. Could you come up with a self-contained example that we could run and experiment with? — NPE
– NPE, Commented Dec 21, 2011 at 15:34
I can run the code (once I've changed nbThread to nbThreads) and I am getting a different error (Exception in thread Thread-4 (most likely raised during interpreter shutdown)). — NPE
– NPE, Commented Dec 22, 2011 at 14:51
Yes, Indeed, my real code is longer. But, in my code, the code below is under an other fuction and I call this function in sub-function from a class... And I call this class with an other file. But It's the same. So I simulate this and I have the same error than you but not every time. I try a print of my arrays, when it's work, it return this kind of things [173[443, [, 917]663, , 931587, , 85, or nothing. That shows than all the Thread works on the same variable. — user1062526
– user1062526, Commented Dec 22, 2011 at 15:31
BTW, I've experimented with your code a bit more, and have been able to completely eliminate scipy from the script, and still get the exception. — NPE
– NPE, Commented Dec 22, 2011 at 15:35

mac · Accepted Answer · 2011-12-21 15:53:35Z

2

Indeed, when several threads are launched, a local array (bootIndexs in the code below) seems to be used for all launched thread.

That's the entire point of threads: lightweight tasks that share everything with their spawning process! :) If you are looking for a share-nothing solution, than you should perhaps look at the multiprocessing module (keep in mind spawing a process is much heavier on the system than spawning a thread, though).

However, back to your problem... mine is little more than a shot in the dark, but you could try to change this line:

boot = [distribModules[i] for i in bootIndexs]

to:

boot = [distribModules[i] for i in bootIndexs.copy()]

(using a copy of the array rather than the array itself). This seems unlikely to be the issue (you are just iterating over the array, not actually using it), but is the only point I can see when you use it in your thread so...

This of course works if your array content is not to be changed by the threads manipulating it. If changing the value of the "global" array is the correct behaviour, then you should contrarily implement a Lock() to forbid simultaneous access to that resource. Your threads should then do something like:

lock.acquire()
# Manipulate the array content here
lock.release()

edited Dec 21, 2011 at 15:53

answered Dec 21, 2011 at 15:17

mac

43.2k27 gold badges126 silver badges133 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user1062526 Over a year ago

I try the copy() but it's the same. I think the trouble is when the array is created (with scipy, in the line above). This array is probably created with a Thread and an another Thread, launched at the same time, try to created an array other the same array... Multiprocessing is harder with python on windows, I feel...

mac Over a year ago

@user1062526 - Reading your comment to the question: the fact you get the error intermittently is a clear sign that it is an error linked to race conditions. I'm surprised - though - that the array is accessed by various threads, as bootIndex is referenced only within the function, so its scope should be unambiguously local... :-/

Josef · Accepted Answer · 2011-12-23 20:17:58Z

1

I have no experience with threading, so this might be completely off the mark.

scipy.stats.randint, as the other distributions in scipy.stats, is an instance of the corresponding distribution class. This means that every thread is accessing the same instance. During the rvs call an attribute _size is set. If a different thread with a different size accesses the instance in the meantime, then you would get the ValueError that the sizes don't match in the reshape. This sounds like the race condition to me.

I would recommend to use numpy.random directly in this case (this is the call in scipy.stats.randint)

numpy.random.randint(min, max, self._size)

maybe you have better luck there.

If you need a distribution that is not available in numpy.random, then you would need to instantiate new instances of the distribution in each thread, if my guess is correct.

answered Dec 23, 2011 at 20:17

Josef

23.1k3 gold badges60 silver badges73 bronze badges

2 Comments

user1062526 Over a year ago

Indeed, I need other distribution sometimes so... The better way, I think, it's to instantiate local variables for each thread... But I didn't succed. I try threading.locals, I try to pass empty array in args for bootIndexs but nothing works.

Josef Over a year ago

if you use spstats.randint (or any of the other scipy.stats distributions) in different threads, they still all refer to the same instance in scipy.stats, independent of whether the other variables or local or not.

Collectives™ on Stack Overflow

threading with python : local variables trouble

2 Answers 2

2 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related