Python Multiprocessing doesn't seem to end

Question

I have a process which moves lots of data into a database. I use multiprocessing for this.

It runs nice and quickly, but even when it's finished (all the rows are moved), it doesn't seem to end.

I've added join as I thought it means that the process won't terminate until all other processes are complete

Is there something I have missed here? why doesn't it end?

p=mp.Pool(cpu_count())
p.map(do_process, result)
p.close()
p.join()

I've noticed that multiprocessing tends to swallow exceptions without reporting them. Try wrapping the body of do_process in a try/except block and printing any exceptions. — Thomas
– Thomas, Commented Jul 8, 2022 at 7:31

Pieter Geelen · Accepted Answer · 2022-07-08 07:28:10Z

2

So a nicer way of doing it, it to use joblib:

import joblib

with joblib.parallel_backend('loky'):
    results = joblib.Parallel(n_jobs=-1)(
        joblib.delayed(do_process)(item) 
        for item in result
        )

I am assuming here that that the result object is mapped over the do_process function. The with statement ensure closing the loop. If you want to know of what is happening, then you can add a verbosity setting to the joblib.Parralel object.

Hope it helps

answered Jul 8, 2022 at 7:28

Pieter Geelen

5482 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

kikee1222 Over a year ago

Thanks a lot - what does 'loky' mean here/

Pieter Geelen Over a year ago

loky is the backend. It is the default backend to spawn more processes and run stuff in parallel. There are backends like dask, that support cluster computers and various GPU accelerated backend. The TL;DR is, loky is default, and probably the fastest on a local machine :)

Collectives™ on Stack Overflow

Python Multiprocessing doesn't seem to end

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related