I have a class variable declared as a list that I want to update from a method declared within that class. However since this method processes a large amount of data, I am using multiprocessing to invoke it and hence I need to put lock on the class variable before updating it. I am unable to figure out how to put such a lock and update the class variable. If it matters, I am only creating one object of the said class at any given time.
Add a comment
|
2 Answers
Because of python's GIL, multiprocessing can only be used whith completely separate tasks, and no shared memory. But you still can make it happend by using multiprocessing shared Array/Value:
from https://docs.python.org/2/library/multiprocessing.html#sharing-state-between-processes
from multiprocessing import Process, Value, Array
def f(n, a):
n.value = 3.1415927
for i in range(len(a)):
a[i] = -a[i]
if __name__ == '__main__':
num = Value('d', 0.0)
arr = Array('i', range(10))
p = Process(target=f, args=(num, arr))
p.start()
p.join()
print num.value
print arr[:]
Now as you asked, you need to ensure that differents processes won't access the same variable at the same time, and use a Lock. Hopefuly, all the shared variable available in the multiprocessing module are paired with a Lock.
To access the lock :
num.acquire() # get the lock
# do stuff
num.release() # don't forget to release it
I hope this helps.
1 Comment
bawejakunal
thanks for pointing me to the documentation. I have used the Manager object to share data between processes. Thanks
If you're using the multiprocessing module (as opposed to multithreading, which is different), then unless I'm mistaken, the multiple processes forked don't share memory and each process would have its own copy of your class. This would mean that a lock would not be necessary, but it would also mean that the class attribute is not shared like you want it to be.
The multiprocessing module does offer several ways to allow communication between processes, including shared array objects. Perhaps this is what you're looking for.
Depending on what you're doing, you might also consider using the master-worker pattern, where you create a worker class with methods to manipulate your data, spawn several processes to run this class, and then dispatch datasets to the workers from your main process using the Queue class from the multiprocessing module.
