Python Multiprocessing Module

Question

I have this script but It returns an error. The problem is that I omit to write

if _name_ == "__main__":

This is the script I took it from a tutorial but Im adapting it to my needs

import concurrent.futures
import time

start = time.perf_counter()


def do_something(seconds):
    print(f'Sleeping {seconds} second(s)...')
    time.sleep(seconds)
    return f'Done Sleeping...{seconds}'


with concurrent.futures.ProcessPoolExecutor() as executor:
    secs = [5, 4, 3, 2, 1]
    results = executor.map(do_something, secs)

    # for result in results:
    #     print(result)

finish = time.perf_counter()

print(f'Finished in {round(finish-start, 2)} second(s)')

the problem I have is that I'm using Windows and I don't know where I have to write: if __name__ == "__main__":

You have to put any function right after if _name_ == "__main__": — KingDarBoja
– KingDarBoja, Commented Aug 26, 2020 at 20:32
You have to put that so that it encompasses the use of the executor, so right after your definition of def do_something — juanpa.arrivillaga
– juanpa.arrivillaga, Commented Aug 26, 2020 at 20:34
Take a look at this example: stackoverflow.com/questions/53924018/… Cheers! — KingDarBoja
– KingDarBoja, Commented Aug 26, 2020 at 20:35
Its too bad that the tutorial ignores something so basic to cross platform coding. If there is a way to provide feedback to the author, it would be good to mention it. — tdelaney
– tdelaney, Commented Aug 26, 2020 at 20:44
@tdelaney The multiprocessing documentation states: "Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process)." — Roland Smith
– Roland Smith, Commented Aug 26, 2020 at 20:50

tdelaney · Accepted Answer · 2020-08-26 20:40:34Z

Your code needs to be import safe because multiprocessing will re-import it in the subprocess. That means that you shouldn't do anything at module level that you don't want to run on mere import. For instance, creating the executor at module level means you'd have an infinite number of subprocesses created because each new one would import the module and create yet another subprocess.

Here's how you would solve the problem

import concurrent.futures
import time

def do_something(seconds):
    print(f'Sleeping {seconds} second(s)...')
    time.sleep(seconds)
    return f'Done Sleeping...{seconds}'

if __name__ == "__main__":
    start = time.perf_counter()
    with concurrent.futures.ProcessPoolExecutor() as executor:
        secs = [5, 4, 3, 2, 1]
        results = executor.map(do_something, secs)

    # for result in results:
    #     print(result)

    finish = time.perf_counter()

    print(f'Finished in {round(finish-start, 2)} second(s)')

bitranox · Accepted Answer · 2020-08-26 20:44:05Z

here the working, with inline comments - You need to prevent that main() is called again when forking-like behaviour under windows.

This is only necessary under windows, because windows does not support fork - therefore python simulates some "forking-like" behaviour and tries to create the same environment in the new process :

import concurrent.futures
import time

def do_something(seconds):
    print(f'Sleeping {seconds} second(s)...')
    time.sleep(seconds)
    return f'Done Sleeping...{seconds}'

def main():
    start = time.perf_counter()
    with concurrent.futures.ProcessPoolExecutor() as executor:
        secs = [5, 4, 3, 2, 1]
        results = executor.map(do_something, secs)

    # for result in results:
    #     print(result)

    finish = time.perf_counter()
    print(f'Finished in {round(finish-start, 2)} second(s)')

if __name__ == '__main__':
    # this guards main() when forking
    main()

Collectives™ on Stack Overflow

Python Multiprocessing Module

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related