Multiprocessing Scipy optimization in Python

Question

I'm trying to run an optimization problem in parallel. The code works well when not in parallel but I struggle to add the multiprocessing layer to it. It's a sort of a vectorized MLE estimation, but I wrote a much simpler code below featuring the same error.

from scipy import optimize
import multiprocessing as mp

# function 'func' to be minimized (with a number of argument>2)
def func(x, arg1, arg2, arg3):
    
    x = x*x + arg1*x + arg2*x + arg3*x
    
    return x

# function 'fit' that is called to minimize function 'func'
def fit(func, arguments):
    
    x0, arg1, arg2, arg3 = arguments

    results = optimize.minimize(func, x0,  args=(arg1, arg2, arg3), method='BFGS')
    
    print(f'value of the function at the minimum: {results.fun}')
    print(f'value of the parameter x when the function is at the minimum: {results.x}')
    
    return results

# main code
if __name__ == "__main__":
    
    # Arbitrary values of the parameters
    x0=100
    arg1=1
    arg2=2
    arg3=3
    
    # gather in a tuple
    arguments=(x0, arg1, arg2, arg3)
    
    # if not run with multiprocessing:
    #fit(func, arguments)

    # multiprocessing
    with mp.Pool(mp.cpu_count()) as pool:
        pool.map(fit,arguments)

The error I get is:

Process SpawnPoolWorker-3:
Traceback (most recent call last):
  File "C:\ProgramData\anaconda3\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "C:\ProgramData\anaconda3\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\ProgramData\anaconda3\lib\multiprocessing\pool.py", line 114, in worker
    task = get()
  File "C:\ProgramData\anaconda3\lib\multiprocessing\queues.py", line 358, in get
    return _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'fit' on <module '__main__' (built-in)>

Another thing I am trying to do is to see at each iteration of the minimization problem the values of results.fun and results.x to know where the algorithm is. I've understood that this is done through callback functions, however I've seen that these can be used with a pool.apply_async function and I'm not sure that it would work for a maximum likelihood estimation problem.

For reference, I'm on Windows and Python 3.8.10

Many thanks for your help!

I'm not at a computer right now to test the multiprocessing, but for observing the progress, have you tried scipy.optimize.minimize(..., options={"disp":True})? — jared
– jared, Commented Jun 10, 2023 at 0:42
I ran your code and did not get the same error, though I did have to adjust the code to get it to run for me. — jared
– jared, Commented Jun 10, 2023 at 1:27
you minimally need to re-work arguments and change fit to take a single argument. (map only supports functions taking a single argument). The single argument can itself be a tuple which you unpack within the function. — Aaron
– Aaron, Commented Jun 10, 2023 at 19:34
That's what I had to do to get Jessica's code working. I added the function to arguments and then changed the fit function signature. — jared
– jared, Commented Jun 10, 2023 at 19:51

Shankar ARUL · Accepted Answer · 2023-06-21 21:12:37Z

0

Try changing arguments to a tuple packed inside a list and it works like a charm !

arguments= [(x0, arg1, arg2, arg3)]

Take care and stay classy ma chère !

edited Jun 21, 2023 at 21:12

answered Jun 21, 2023 at 15:48

Shankar ARUL

14k14 gold badges74 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Jessica F. Over a year ago

Thanks Shankar. This now works if I try on Noteable.io but not on my laptop. It's weird, I'm trying to figure out why.

Jessica F. · Accepted Answer · 2023-06-23 16:25:38Z

Thanks everyone for your help. This works on Pycharm. It seems to be a Spyder version issue!!

from scipy import optimize
import multiprocessing as mp

# function 'func' to be minimized (with a number of argument>2)

def func(x, arg1, arg2, arg3):
    
    x = x*x + arg1*x + arg2*x + arg3*x

    return x

# function 'fit' that is called to minimize function 'func'
def fit(arguments):
    
    x0, arg1, arg2, arg3 = arguments
    
    results = optimize.minimize(func, x0,  args=(arg1, arg2, arg3), method='BFGS', options={"disp":True})
    
    print(f'value of the function at the minimum: {results.fun}')
    print(f'value of the parameter x when the function is at the minimum: {results.x}')
    
    return results

# main code
if __name__ == "__main__":
    
    # Arbitrary values of the parameters
    x0=100
    arg1=1
    arg2=2
    arg3=3
    
    # if not run with multiprocessing:
    # arguments= (x0, arg1, arg2, arg3)
    # fit(arguments)

    # multiprocessing
    arguments= [(x0, arg1, arg2, arg3)]
    with mp.Pool(mp.cpu_count()) as pool:
        pool.map(fit,arguments)

Collectives™ on Stack Overflow

Multiprocessing Scipy optimization in Python

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related