Exception in thread QueueManagerThread - scikit-learn

Question

When I set n_jobs=-1 I get error and if I set n_jobs equal big value (n_jobs=100), but if set smaller value (e.g. n_jobs=32), it works fine. I've tried reinstall scikit-learn and joblib packages, but to no avail. Also, it (n_jobs=-1) works fine previously, but suddenly go wrong.

from sklearn import datasets
from sklearn.model_selection import cross_validate, StratifiedKFold
from sklearn.linear_model import RidgeClassifier

iris = datasets.load_iris()
iris_X = iris.data
iris_y = iris.target

skf = StratifiedKFold(n_splits=5, random_state=0, shuffle=True)
scoring = {'accuracy': 'accuracy'}

model_ridge = RidgeClassifier(random_state=0)
scores = cross_validate(estimator=model_ridge, 
                            X=iris_X,
                            y=iris_y,
                            scoring=scoring,
                            cv=skf.split(iris_X, iris_y),
                            n_jobs=-1)

Exception in thread QueueManagerThread:

Traceback (most recent call last): File "C:\ProgramData\Anaconda3\envs\py36\lib\threading.py", line 916, in _bootstrap_inner self.run()

File "C:\ProgramData\Anaconda3\envs\py36\lib\threading.py", line 864, in run self._target(*self._args, **self._kwargs)

File "C:\ProgramData\Anaconda3\envs\py36\lib\site-packages\sklearn\externals\joblib\externals\loky\process_executor.py", line 615, in _queue_management_worker ready = wait(readers + worker_sentinels)

File "C:\ProgramData\Anaconda3\envs\py36\lib\multiprocessing\connection.py", line 859, in wait ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)

File "C:\ProgramData\Anaconda3\envs\py36\lib\multiprocessing\connection.py", line 791, in _exhaustive_wait res = _winapi.WaitForMultipleObjects(L, False, timeout)

ValueError: need at most 63 handles, got a sequence of length 65

This problem origin from joblib/loky. I created the issue on github. — Kirill Dolmatov
– Kirill Dolmatov, Commented Jul 24, 2019 at 8:50

Baz · Accepted Answer · 2019-07-24 04:46:16Z

1

Sounds like you're running a lot of cores there! Set n_jobs=60 and this should work. See more details here https://github.com/psf/black/issues/564

answered Jul 24, 2019 at 4:46

Baz

5283 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Kirill Dolmatov Over a year ago

This may work, but it ins't solution of this problem, because I would like to utilize cpu power completely. With similar success, I could dont use parallel computation at all.

Mark Dabler Over a year ago

Check out dask!

Collectives™ on Stack Overflow

Exception in thread QueueManagerThread - scikit-learn

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related