92 questions
1
vote
1
answer
79
views
What are the basic rules for propagating lists and dictionaries across processes using the Manager of the multiprocessing module?
I am trying to use the multiprocessing module to parallelize a CPU-intensive piece code over multiple cores. This module looks terrific in several respects, but when I try to pass lists and ...
- 427
1
vote
0
answers
107
views
How to Efficiently Use Python's Process Manager with Multiple Processes?
My problem is part of a very large and complicated project. I don't want to share all the details here as it's too much, but I want to describe the main parts in an example program.
I have a large ...
- 23
0
votes
0
answers
99
views
Python Multiprocessing with mp.Manager().dict() is giving runtime error when constructed using class
I have been researching but couldn't solve this one with the lack of documentation on multiprocessing.Manager().dict(). I'm trying to download tickers information from yahoo finance using ...
1
vote
1
answer
789
views
Python Multiprocessing with pydantic.BaseModel objects
I'm trying to process a pydantic object with the multiprocessing toolbox in Python.
My Task:
I need to download many files. The url to these files an additional information are stored in an data ...
- 43
0
votes
0
answers
103
views
How to maintain a pool of processes
I have a project in which I run multiple processes to do multiple jobs, the jobs are distributed via a queue.
queue = multiprocessing.Queue()
process = multiprocessing.spawn(
run_alg,
args=(...
- 1,839
-1
votes
1
answer
365
views
How to share in memory resources between different workers while using Flask with Gunicorn
I have a flask App. I am creating 4 workers with Gunicorn. So, on running this,
I will have total 4 flask instances, 4 PIDs and 4 workers. I want to use a python dictionary across all these workers.
I ...
- 98
0
votes
1
answer
457
views
How to limit the number or parallel executions using Python multiprocessing - Queue(), Manager(), Pool()
Hej all,
I am struggling to limit the number of parallel executions in the below Python code using multiprocessing - in particular Queue(), Manager() and Pool().
My understand was that multiprocessing....
- 3
0
votes
1
answer
93
views
Python - Troubles sharing objects between processes using Manager.list
I am trying to share some images between two Python processes using a Manager().list() object.
I have a main file, where I create and launch the processes:
main.py
if __name__ == "__main__":
...
- 1,630
0
votes
0
answers
258
views
Why is reading values from a shared dictionary created with Python multiprocessing's manager much slower than reading from disk
I'm using Python's multiprocessing pool and I created a dictionary using a manager, which the pool will read values from. However, I have encountered an issue where reading values from this dictionary ...
1
vote
1
answer
238
views
Multiprocessing concurrent.futures not update global dict and list
I need your help with multiprocessing to update global dict data.
I use two functions, the first function is to do multiprocessing in the for loop.
the second function is to update global dict data.
...
- 11
0
votes
1
answer
102
views
f-string and multiprocessing.Manager dict and list
I am writing this question, because it does not seem like what I am experiencing should be the desired behavior:
The following code is pretty much lifted from the 3.9.16 documentation on ...
- 3,737
1
vote
1
answer
324
views
How to use multiprocessing Value to pass image base64 string?
on a project of mine I try to run Figure.savefig() on seperate multiprocessing.Process (due memory leak) but I am don't manage to allocate shared variable long enough for passing the image string.
See ...
- 45
0
votes
1
answer
450
views
Python sharing multiprocessing.Manager.list with child processes
My question is so simple. I want to share a multiprocessing.Manager().dict() with child processes. But shared dict will be initialized AFTER starting childs. Here is the example code:
import ...
- 144
0
votes
0
answers
148
views
pass object of selenium from a subprocess to another subprocess
I try to use manager and queue to share the object of selenium in two subprocess, but it shows me the error "AttributeError: Can't pickle local object '_createenviron..encodekey'"
from ...
- 1
1
vote
1
answer
262
views
How to access property attribute from multiprocess Proxy class in Python?
I am trying to get attribute from my Proxy class but I don't quite understand the implementation (as per https://docs.python.org/3.9/library/multiprocessing.html?highlight=multiprocessing#...
2
votes
2
answers
829
views
Python Multiprocessing custom manager with associated objects
I'm trying to make a class object usable for multiple processes. Unfortunarely this seems to be more of an issue than I anticipated.
I have the following class object:
class BusObject:
inputs: ...
- 202
2
votes
1
answer
441
views
Update shared dictionary using mpire package
I am working to update a shared dictionary synchronously using mpire package in Python in a multi-core machine (i.e., parallel processing to update a dict). The environment I am using is a Linux ...
- 121
0
votes
0
answers
58
views
Bug or error when attempting to append process to list/array and return
When attempting to append process id to list inside processing_task the list is changed but when attempting to return it from function it is unchanged for some reason I don't understand what the ...
- 97
2
votes
1
answer
889
views
Is multiprocessing.Manager.Queue() thread and process safe?
I am fairly new to multiprocessing in python. I am trying to use Queues to pass messages between processes. As I read through the documentation it says multiprocessing.Queue() is thread and process ...
1
vote
0
answers
162
views
python multiprocessing.managers and Flask - how long is data stored?
I have a Flask app that generates output based on a query, and gives user multiple options to do more with the output. I need to store the data somewhere between requests and the best solution I found ...
-1
votes
1
answer
773
views
How to run parallel processes with multiprocessing terminate one of them from another and then restart
I get error when running the code
TypeError: Pickling an AuthenticationString object is disallowed for
security reasons
I am very new to this... You can add your own completely new solution/code to ...
- 97
0
votes
0
answers
167
views
What makes Python Multiprocessing raise different errors when sharing objects between processes?
Context: I want to create attributes of an object class in parallel by distributing them in the available cores. This question was answered in this post here by using the python Multiprocessing Pool.
...
- 67
0
votes
1
answer
201
views
Accessing multiprocessing manager dict from webserver
I've been trying to share a multiprocessing manager dict with a web server so far with little success. The dict contains information but somehow it is not propagated to the running web server. The ...
- 55
1
vote
1
answer
1k
views
ValueError: 'Pool not running' when trying to loop the multiprocessing pool
I'm trying to run the same pool three times so that it adds the values in the same list:
import multiprocessing
def foo(name,archive):
print('hello ', name)
archive.append(f"hello {name}&...
- 2,222
1
vote
1
answer
161
views
Even using Manager.list() the values are not being appended to the list using multiprocessing pool
As can see in the code below, I'm generating a multiprocessing with manager and list comprehension to handle the pool:
import multiprocessing
def foo(name,archive):
print('hello ', name)
...
- 2,222
0
votes
1
answer
187
views
Unreliable multiprocessing Manager
I was having a problem with some more complex code but have reduced it to something that reproduces the issue:
from multiprocessing import Manager
from concurrent.futures import ProcessPoolExecutor
...
- 29.1k
1
vote
1
answer
2k
views
Changing values of list in multiprocessing
I am new to python multiprocessing, a background about the below code. I am trying to create three processes, one to add an element to the list, one to modify element in the list, and one to print the ...
-1
votes
1
answer
136
views
Pool apply function hangs and never executes
I am trying to fetch Rally data by using its python library pyral. Sequentially the same code works, but its slow.
I thought of using python multiprocess package, however my pool.apply method gets ...
- 59
1
vote
1
answer
1k
views
Sharing and Updating List between Processes and Threads
Trying to Share AND Update a list between Main Process - Main Thread & Sub Process - Sub Thread. Currently it appears that both the main process and main thread share the update of one copy of the ...
0
votes
1
answer
1k
views
Python multiprocessing can't use functions from other module
Update: it's working after updating my Spyder to 5.0.5. Thanks everyone!
I am trying to speed up a loop using multiprocessing. The code below aims to generate 10000 random vectors.
My idea is to split ...
- 133
1
vote
1
answer
3k
views
Sharing a list between processes with multiprocessing.Manager
1st
'''
import multiprocessing as mp
def func( list):
x = 5
list = list*5
print('list_in_sub: ', list)
if __name__ == "__main__":
list = mp.Manager().list(range(2))
p1 ...
- 11
0
votes
1
answer
130
views
How to share complex objects in multiprocessing manager
I know this question is being asked in many threads but so far I couldn't find proper solution.
Below is my code:
Class A:
def init(self):
self.b = B()
Class B:
def init(self):
.......
- 874
1
vote
2
answers
987
views
Sharing asyncio objects between processes
I am working with both the asyncio and the multiprocessing library to run two processes, each with one server instance listening on different ports for incoming messages.
To identify each client, I ...
- 947
1
vote
1
answer
864
views
How do read and writes work with a manager in Python?
Sorry if this is a stupid question, but I'm having trouble understanding how managers work in python.
Let's say I have a manager that contains a dictionary to be shared across all processes. I want to ...
- 1,107
0
votes
1
answer
81
views
Python multiprocessing manger.list() not being pass to Matplotlib.animation correctly
I have this 2 processes with a list created with manager.list() being shared between them, one is called DATA() and it is "generating" data and appending to the list and the other is ...
- 57
0
votes
1
answer
743
views
Change the spawned process name for Python multiprocessing manager
I want to change the process name of the spawned multiprocessing's manager SyncManager using setproctitle, I tried extending multiprocessing.managers.SyncManager:
class MySyncManager(multiprocessing....
- 377
0
votes
1
answer
1k
views
Python IPC Shared Memory between two scripts
I am trying to develop an application for the Raspberry Pi3. In fact 2 applications. Application 1 will be a simple gpio read, count the number of times a particular gpio input goes high and log into ...
- 49
0
votes
0
answers
128
views
The best way to share a class between processes
First of all, I'm pretty new in multiprocessing and I'm here for learning of all of you. I have several files doing something similar to this:
SharedClass.py:
class simpleClass():
a = 0
b =...
- 389
1
vote
1
answer
834
views
How to use Python3.9.2 multiprocessing.SyncManager among remote processes?
I have modeled the following test program on the relevant Python 3.9.2 documentation regarding the synchronization of data among remote processes. As nearly as I can tell, though, it doesn't actually ...
- 208
0
votes
0
answers
311
views
The read speed of Manager.dict() is too slow
I have a huge dict variable which is about 2 Gigabytes.
I am doing some scientific calculation on this dict(read only). However, the reading speed of the shared dictionary is much much slower than a ...
0
votes
1
answer
874
views
Python multiprocessing - Using shared variables with a manager between threads?
I am using a Manager from pythons multiprocessing library. I would like to share variables between threads.
So I have a function and this function uses one such variable internally. Because of python'...
0
votes
0
answers
139
views
Multiprocessing manager.list using list subclass
I am trying to use my list subclass on calling manager.list. So, I can add my own methods, but it is not working. Is it possible to do this?
class mylist(list):
def __init__(self, *args):
...
- 455
0
votes
0
answers
248
views
multiprocessing.Process does not do anything
I am trying to understand parallel processing using multiprocessing in python using the example given in this link https://pymotw.com/2/multiprocessing/basics.html.
Here is the code:
def worker(i):
...
- 164
0
votes
0
answers
396
views
How to speed up multiprocessing manager?
I created my own manager since I will eventually need to pass classes into different processes. I did this using the following code.
class MyManager(BaseManager):
pass
MyManager.register('modeling',...
- 65
0
votes
0
answers
49
views
Python multiprocessing: distribute tasks to managers asynchroneously
I need to hold instances of python objects in different processes, call methods of these objects and collect the results asynchroneously. My approach was to create a manager for each process and ...
- 2,740
1
vote
1
answer
653
views
Multiprocessing proxy: let getters return proxies themselves
I have a complex unpickable object that has properties (defined via getters and setters) that are of complex and unpickable type as well. I want to create a multiprocessing proxy for the object to ...
- 2,740
2
votes
1
answer
3k
views
How to properly share Manager dict between processes
What I would like to do is to share a dictionary between subclasses of Process and when one process updates the dictionary the other is notified to use it. This is illustrated in the code below where ...
- 41
1
vote
1
answer
649
views
How to access classes stored in a multiprocessing.Manager list?
I can access and change the attributes of classes in a list like this:
class TestClass:
def __init__(self):
self.value = 1
instances = [TestClass()]
instances[0].value = 42
print(...
- 426
1
vote
1
answer
1k
views
Python Multiprocessing: How to make Pool and Manager work together?
I am trying to understand how Pool() and Manager() can be combined in Python's multiprocessing library for having shared objects between the individual worker processes. In my example, I would like to ...
- 196
2
votes
0
answers
657
views
How to construct proxy objects from multiprocessing.managers.SyncManager?
I have long running file I/O tasks which I'd like to be able to move into a daemon/server process. A CLI tool would be used to queue new jobs to run, query the status of running jobs, and wait for ...
- 797