191 questions
0
votes
1
answer
181
views
Python ThreadPoolExecutor(max_workers=MAX_PARALLEL_REQUESTS) asyncio analog
When I use ThreadPoolExecutor, I can send a requests batch with limitation of parallel requests like this:
with ThreadPoolExecutor(max_workers=MAX_PARALLEL_REQUESTS) as pool:
results = list(pool....
- 1,409
0
votes
1
answer
508
views
How do I get ipyparallel to print status updates while in wait_interactive() in Jupyterlab?
I have the following code blocks, each in JupyterLab (this works as expected in Jupyter, but not Jupyter-Lab):
def work(sleepTime):
import time
import datetime
start = datetime.datetime....
- 26.3k
0
votes
1
answer
555
views
AttributeError: module 'ipyparallel' has no attribute 'Cluster'
I am going through the tutorial to learn ipyparallel and while doing so, I got the error: AttributeError: module 'ipyparallel' has no attribute 'Cluster'
I uninstalled and reinstalled the package but ...
- 1
1
vote
0
answers
142
views
IPyparallel: cannot remove cluster
I'm trying to use the function remove_cluster of ipyparallel.ClusterManager()
import ipyparallel as ipp
cluster = ipp.Cluster(n=2)
# start cluster syncronously
cluster.start_cluster_sync()
# <...
- 29.7k
4
votes
1
answer
4k
views
ipykernel_launcher processes are consuming memory, Not able to kill
What are these zombie ipykernel_launcher process in my machine, which are hogging to much memory:
This is output of htop command, but I ps for those processes,(to kill them) I do not see them as:
ps ...
- 656
0
votes
1
answer
147
views
How to use `client.start_ipython_workers()` in dask-distributed?
I am trying to get workers to output some information from their ipython kernel and execute various commands in the ipython session. I tried the examples in the documentation and the ipyparallel ...
- 16.7k
1
vote
2
answers
5k
views
How do I use multiprocessing on Python to speed up a for loop?
I have this code which I would like to use multi-processing to speed up:
matrix=[]
for i in range(len(datasplit)):
matrix.append(np.array(np.asarray(datasplit[i].split()),dtype=float))
The ...
- 31
2
votes
1
answer
621
views
ipyparallel: what is the most reliable way to clear/reset engine namespaces without restarting the cluster?
As far as I understand, an ipython cluster manages a set of persistent namespaces (one per engine). As a result, if a module that is imported by an engine engine_i is modified, killing the main ...
- 4,778
2
votes
1
answer
1k
views
Can't run IPython-Parallel in Jupyter Notebook
I'm trying to follow these instructions on using ipyparallel in order to speed up some Python scripts in Jupyter Notebook. When doing
import ipyparallel as ipp
IPPC = ipp.Client()
I get the following ...
- 21
2
votes
2
answers
756
views
IPython-Parallel: ipcluster and ipcontroller-client.json not found
I'm trying to follow these instructions on using ipyparallel in order to speed up some Python scripts in Jupyter Notebook. When doing
import ipyparallel as ipp
IPPC = ipp.Client()
I get the following ...
- 21
10
votes
2
answers
476
views
Large memory consumption by iPython Parallel module
I am using the ipyparallel module to speed up an all by all list comparison but I am having issues with huge memory consumption.
Here is a simplified version of the script that I am running:
From a ...
- 101
1
vote
0
answers
49
views
Parallelizing for loops in python
I know similar questions on this topic have been asked before, but I'm still struggling to make any headway with my problem.
Basically, I have three dataframes (of sizes 402 x 402, 402 x 3142, and 1 ...
- 11
1
vote
1
answer
1k
views
joblib Parallel running out of memory
I have something like this
outputs = Parallel(n_jobs=12, verbose=10)(delayed(_process_article)(article, config) for article in data)
Case 1: Run on ubuntu with 80 cores:
CPU(s): 80
...
- 1,579
1
vote
1
answer
286
views
Nested parallelism with scikit learn models
I want to to do nested parallelism with scikit learn logisticregressionCV inside a for loop:
for i in range(0,10):
logisticregressionCV(n_jobs=-1)
I want to parallelize the for loop as well.
I ...
- 11
1
vote
0
answers
659
views
Parallel file reading in python
I have been trying to read a large file and writing to another file at the same time after processing the data from the input file, the file is pretty huge around 4-8 GB, is there a way to parallelise ...
- 313
3
votes
0
answers
412
views
How print from ipyparallel engine to the main process?
I implemented a parallel function using the ipyparallel library. I would like to ask you if there is a way to print from the engine to the main process? Here what I'm trying to do:
import ipyparallel ...
0
votes
1
answer
3k
views
Ipython Widgets (how to make a timer)
I made this timer widget that works the way I want, but it locks up the notebook so I can't execute other code cells at the same time.
Here's an example of what I mean:
Any ideas for getting the ...
- 815
3
votes
1
answer
1k
views
ipyparallel Parallel for loop that appends to list
I have a loop which does a bunch of CPU intensive calculations and appends the result to a list, per iteration.
How can I have that working in parallel. On C# there are concurrent containers, how ...
- 430
3
votes
0
answers
478
views
Use ipython widgets to manipulate plots, that are constantly updated from within an infinite loop
Within an ipython notebook, using Jupyter I try to do a calculation, thats running for an extended period of time in a while loop. I use pyplot to show the current status of my calculation and I would ...
- 31
1
vote
1
answer
271
views
Specify a number of ipengine instances to be launched within ipyparallel cluster
Speaking of ipyparallel, is it possible to specify a number of ipengines to simultaneously launch on a slave machine, and if so - how do I do it?
For example, one can specify a number of engines to ...
- 2,554
1
vote
0
answers
203
views
Optimization: alternatives to passing large array to map in ipyparallel?
I originally wrote a nested for loop over a test 3D array in python. As I wanted to apply it to larger array which would take a lot more time, I decided to parallelise using ipyparallel by writing it ...
- 45
1
vote
0
answers
424
views
Converting Ipyparallel to Dask - Where is the Web UI
I have set up a cluster with ipyparallel on my local machine and wish to convert it to a dask cluster. My primary motivation for this is to be able to use the web ui with dask (via bokeh) to monitor ...
- 462
4
votes
2
answers
2k
views
Starting ipcluster from code
I want to dynamically start clusters from my Jupyter notebook for specific functions. While I can start the cluster and get the engines running, I am having two issues:
(1) I am unable to run the ...
- 1,432
6
votes
0
answers
524
views
launching ipyparallel cluster across multiple nodes using MPI
I am trying to start a ipyparallel cluster using MPI.
The ipcluster_config has following lines modified as such:
c.MPILauncher.mpi_cmd = ['mpiexec']
c.MPIControllerLauncher.controller_args = ['--...
- 2,065
1
vote
0
answers
127
views
Specify custom kernel when using ipcluster
I have a custom Jupyter kernel. It is specified using a kernel.json kernelspec.
An extract from kernel.json shows how to start the custom kernel:
source /opt/conda/bin/activate gdb_ipykernel &&...
- 966
6
votes
1
answer
4k
views
Loading lightgbm model and using predict with parallel for loop freezes (Python)
I have the need to use my model to do predictions in batches and in parallel in python. If I load the model and create the data frames in a regular for loop and use the predict function it works with ...
- 846
3
votes
0
answers
361
views
ipyparallel won't let me push an object to an engine
Trying out ipyparallel, and getting stuck on the very early step of pushing code to my engines. Started up my engines with:
ipcluster start -n 4
Then try out putting things on the engines with:
...
- 995
1
vote
0
answers
199
views
debugging ipyparallel applications
Thinking of using ipyparallel to develop a machine learning algorithm on a cluster, mainly using pandas, scikit-learn and numpy.
What are the recommended debugging techniques. Is it possible to ...
- 11
1
vote
0
answers
603
views
Is ipython or numpy secretly parallelizing matrix multiplication?
So the case is the following:
I wanted to compare the runtime for a matrix multiplication
with ipython parallel and just running on a single core.
Code for normal execution:
import numpy as np
n = ...
- 157
2
votes
0
answers
341
views
ipyparallel's LoadBalancedView bloats memory, how can I avoid that?
This issue may be related to https://github.com/ipython/ipyparallel/issues/207 which is also not marked as solved, yet.
I also opened this issue here https://github.com/ipython/ipyparallel/issues/286
...
- 297
0
votes
1
answer
335
views
Parallelizing subgraph tasks in Python
This might be a naive question but I've really tried searching multiple resources: multiprocessing and ipyparallel but these seem to be lack of appropriate information for my task.
What I have is a ...
- 620
1
vote
0
answers
239
views
Why can't ipyparallel push/pull when using cloudpickle?
I came across this issue while using ipyparallel. When I try to use cloudpickle, it appears I cannot push or pull globals anymore. Does anybody know the reason or a way around this?
In general, I ...
- 794
3
votes
2
answers
2k
views
ipyparallel Exception: "You have attempted to connect to an IPython Cluster but no Controller could be found"
I've just started using ipyparallel, I'm using VS2017 and importing it as;
import ipyparallel as ipp
And then attempting to start it using;
def main():
rc = ipp.Client()
if __name__ == "__main__"...
- 483
0
votes
0
answers
42
views
Code gets hung when using multiprocessing
I want to run a simple function across seperate cores on my computer. My computer has four cores.
To start with, a simple function:
def exp(x):
return x**2
now I want to give this function ...
- 3,587
3
votes
0
answers
500
views
sync_imports for Ipyparallel with multiple path dependencies
I would like to import a class, lets call it MyClass, and assume it is stored in a file MyClass.py. However, the class itself depends on various files distributed over several folders. So if I want to ...
- 383
0
votes
0
answers
479
views
When I use a sklearn parallel module "from cdnmf_fast import _update_cdnmf_fast", its parallelization is not working
(This is my first time of asking a question, if you think the tags or description have something wrong, please tell me, thank you!)
I'm doing a work on Matrix Factorization and use the module sklearn....
- 31
0
votes
1
answer
783
views
How to import ipyparallel function from user defined package, Python
I have a python script that contains a number of user defined functions that I have set up a as package locally. I can run all of the functions in the input_processing.py script except for the one ...
- 391
1
vote
1
answer
641
views
how are the multiprocessing and threading and thread pooling working
https://code.tutsplus.com/articles/introduction-to-parallel-and-concurrent-programming-in-python--cms-28612
From this link I have studied, I have few questions
Q1 : How thread pool (Concurrent) and ...
- 51
0
votes
1
answer
230
views
How can I set the pythonpath and path of an ipengine (using ipyparallel)?
Using Windows / ipython v6.0.0
I am running ipcontroller and a couple of ipengines on a remote host and all appears to work fine for simple cases.
I try to adjust the pythonpath on the remote host (...
- 85
0
votes
1
answer
210
views
Start IPython Parallel from within another Python script
Suppose I have two Python files
test.py
from ipyparallel import Client
def hi(a):
return b + (a * 2)
def run():
b = 3
client = Client()
view = client[:]
view.push({'b':b})
...
- 77
0
votes
1
answer
80
views
Notification when a new task starts in ipython parallel
A couple of questions related to best practices with ipyparallel. I'm attempting use it to implement a Monte Carlo framework for a model that takes ~15 to run. The idea is to run N engines (via SLURM) ...
- 154
4
votes
0
answers
184
views
Iterate through AsyncMapResult where *some* tasks have raised an exception
When using an ipyparallel cluster to process tasks in parallel, how do I iterate over the AsyncMapResult when some of the tasks have raised an exception?
All I get is the exception, but I can't ...
- 1,729
1
vote
1
answer
300
views
passing different arguments for ipython parallel clients
I am learning parallel computation in ipython. I came across an example,
from ipyparallel import Client
rc = Client()
rc.block = True
print(rc.ids)
def mul(a,b):
return a*b
dview = rc[:]
print(...
- 9
3
votes
0
answers
205
views
How do I avoid restarting iPython parallel clusters every time I restart the Jupyter kernel?
I am running iPython Parallel.
Every time I restart the main notebook kernel, I have to restart the cluster as well. Without doing so, I get a NoEnginesRegistered error (in other words, the ...
- 19.6k
6
votes
0
answers
3k
views
How to parallelize with jupyter and sklearn?
I'm trying to parallelize the GridSearchCV of scikit-learn. It's running on a jupyter (hub) notebook environment. After some research I found this code:
from sklearn.externals.joblib import Parallel, ...
- 5,318
4
votes
2
answers
703
views
Can I restart an iPython cluster from a notebook?
I'm just wondering if there is some python code or magics I can execute that will restart the ipython cluster. It seems like every time I change my code, it needs to be restarted.
- 19.6k
1
vote
0
answers
194
views
How do I debug iPython Parallel engine death?
I've got a cluster of engines running. When I set them to work on a long running calculation, after a couple of minutes they just seem to 'die' silently, one by one, until the calculation stalls ...
- 19.6k
0
votes
1
answer
967
views
ipyparallel with mpi cannot find engines
I'm trying to get a basic ipyparallel environment working using mpi4py as described in the ipyparallel documentation. After starting the ipcluster, I load ipython and try to create a client but it has ...
- 114
0
votes
1
answer
120
views
Implement parallel processing of for loop
Looking to make the following code parallel- it reads in data in one large 9gb proprietary format and produces 30 individual csv files based on the 30 columns of data. It currently takes 9 minutes per ...
- 101
1
vote
1
answer
96
views
Ipyparallel error with directview
When running this code, I have this error :
from ipyparallel import error, AsyncHubResult, DirectView as dv, Reference
@dv.parallel(block=True)
def np_std_par(x):
return np_std(x)
TypeError: ...
- 3,390