How to run a function in python in parallel with the same arguments?

Question

How do I run several independent processes in parallel when the argument is the same? My current (not nice) solution is:

import time
import multiprocessing

def parse_args():
   ...
   return args

def my_function(args):
    ...

if __name__ == '__main__':
    args = parse_args()
    processes = []
    for i in range(5):
        processes.append(multiprocessing.Process(target=my_function, args=(args,)))
        processes[-1].start()
    time.sleep(200)
    for i in range(5):
       processes[i].terminate()

Also, my_function runs infinetely and it doesn't return anything.

What is "not nice" about your current solution? Seems like it should work just fine.. I personally would use a semaphore to signal my_function to quit gracefully rather than terminate-ing it, but if there's no specific need to gracefully shut down, there's no problem with that. A multiprocessing.Pool would just add lots of overhead and complexity you don't seem to be using anyway. — Aaron
– Aaron, Commented Apr 18, 2022 at 18:46

Loïc · Accepted Answer · 2022-04-18 11:45:36Z

1

I'd join them, and make sure they terminate on their own, like so :

processes = [Process(target=my_function, args=(args,), daemon=True) for p in range(nb_processes)] # create nb_processes running my_function
[p.start() for p in processes] # start all processes
[p.join()  for p in processes] # wait for all processes to end

You want to make sure that my_function implements some sort of timeout, because that'll wait for all processes to complete.

As for getting back their results you could use a queue, check multiprocessing.Queue, or a message broker. I personally like to use REDIS for that, but then it's very much opinion oriented.

As a side note, you probably want to take a look at asyncio if you haven't yet.

answered Apr 18, 2022 at 11:45

Loïc

12k2 gold badges35 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

sigmarule69 · Accepted Answer · 2022-04-18 11:28:49Z

0

Simply use threading as shown in this example

import threading as thd

def function_one():
    while True:
        print("alice")

def function_two():
    while True:
        print("bob")

first_thd = thd.Thread(target=function_one)
second_thd = thd.Thread(target=function_two)

first_thd.start()
second_thd.start()

you can call multiple threads at the same time

answered Apr 18, 2022 at 11:28

sigmarule69

341 silver badge2 bronze badges

1 Comment

Mark Tolonen Over a year ago

threading Python code doesn't run them in parallel due to the GIL.

rcshon · Accepted Answer · 2022-04-18 12:00:54Z

0

Use a process pool and map your calls with something like itertools to keep it short and sweet.

import multiprocessing
import itertools

num_processes = 5

with multiprocessing.Pool(num_processes) as pool:        
    pool.map_async(my_function, itertools.repeat(args, num_processes))
    time.sleep(200)
    pool.terminate()

Note: The map_async returns a future object that allows you to call .get() if you ever need the result.

edited Apr 18, 2022 at 12:00

answered Apr 18, 2022 at 11:51

rcshon

9271 gold badge7 silver badges12 bronze badges

1 Comment

georgejefferson2 Over a year ago

I searched for approximately this solution but my_function should generate a folder but if I use this solution it doesn't. I guess the function generates an error but I don't see it. I guess I will try to write a minimal reproducible example and post a new question

Collectives™ on Stack Overflow

How to run a function in python in parallel with the same arguments?

3 Answers 3

Comments

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related