0 
from itertools import product
from multiprocessing import Pool

with Pool(4) as pool:
            pool.map(lambda x: run_test_function(x, arg2, arg3, arg4), arg1)

I am getting below error after executing above code. There some other code as well which I can't write here. But actual problem is coming from this piece of code only.

Traceback (most recent call last):
  File "modProfileChange_test.py", line 347, in <module>
    main(sys.argv[1:])
  File "modProfileChange_test.py", line 336, in main
    test_run_code(arg1, arg2, arg3, arg4, arg5, arg6)
  File "modProfileChange_test.py", line 23, in test_run_code
    with Pool(4) as pool:
AttributeError: __exit__
Improve this question
14
  • This is already syntactically valid Python 2.7 code Commented Aug 17, 2021 at 9:39
  • And what about library used? I am following only your code on another question. So I am not acquainted much with this library. Commented Aug 17, 2021 at 9:41
  • nothing special, I guess? This is valid Python 2.7 code Commented Aug 17, 2021 at 9:42
  • Its throwing error. Commented Aug 17, 2021 at 10:04
  • 1
    the code you posted only has 5 lines, though. Please post a minimal reproducible example and the full error message (which should start with the word "Traceback"). Commented Aug 17, 2021 at 11:11

1 Answer 1

Reset to default
2
  1. In Python 2.7, multiprocessing.Pool is not a context manager and thus it can't be used in a with statement
    1. Solution - create a pool using regular assignment to a variable: 
      my_pool = Pool(4)
my_pool.map(...)
  2. lambda functions don't work with multiprocessing.Pool, even in Python 3.
    1. Solution - emulate a closure using a solution in the link above: 
      from functors import partial

def run_test_function(x, fun_arg2, fun_arg3, fun_arg4):
    # your code here
    ...

process_func = partial(run_test_function, fun_arg2=arg2, fun_arg3=arg3, fun_arg4=arg4)

Putting this together:

from multiprocessing import Pool
from functools import partial

def run_test_function(x, fun_arg2, fun_arg3, fun_arg4):
    # this is an example
    print x, fun_arg2, fun_arg3, fun_arg4

if __name__ == "__main__":
    arg1 = 1,2,3,4
    arg2 = "hello"
    arg3 = "world"
    arg4 = "!"

    process_func = partial(run_test_function, fun_arg2=arg2, fun_arg3=arg3, fun_arg4=arg4)

    my_pool = Pool(4)
    my_pool.map(process_func, arg1)

Output:

~/test $ python2.7 so10.py
1 hello world !
2 hello world !
3 hello world !
4 hello world !
Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

Thank you, it worked for me. What is the range of processors we can give? On what factor this value depends as you have given 4?
If your workloads are CPU-intensive (which is what the multiprocessing module is used for, as opposed to the threading module), it makes sense to create one process per CPU core. I know my CPU has 4 cores, so I created 4 processes. Pool can determine the number of cores automatically, so you can just use Pool() without any arguments
After the execution of script, script worked fine but cli is getting hung, it went unresponsive.
@Vanshika, what's the script? The one in my answer?
Yeah, this is only the example I have used in my script. Moreover output isn't formatted, it is like you getting 1 hello world !, but it should be like 1 \nhello \nworld \n!
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.