How can i execute the same python program in parallel (I am thinking of x10) with the only difference being two input arguments representing a time-range? I need it for some data-processing, which otherwise will take too long to finish
I know i can do it manually in shell by starting 10 scripts one by one, but it does not seem to be the most "elegant" solution + I would also love to define arguments for each of those programs dynamically in the "main" python program.
Is there a way to do it?
Enclose your script in a main method, like so:
def main(args):
a, b = args
# do something
if __name__ == '__main__':
args = parse_arguments()
main(args)
Then you can use a second script together with multiprocessing.Pool, to run the main method with different arguments.
from myscript import main
from multiprocessing import Pool
a = [1, 2, 3, 4, 5]
b = [6, 7, 8, 9, 10]
if __name__ == '__main__':
with Pool(4) as pool: # four parallel jobs
results = pool.map(main, zip(a, b))
Pool
You can try the following in shell:
python main.py [1, 2, 3, 4, 5] &
python main.py [6, 7, 8, 9, 10]
