Python how to use multiprocessing to different functions with return

Question

I have several 3 small functions:

def getnpx(mt, age, interest):
    val = 1
    initval = 1
    for i in range(age, 55):
        val = val * mt[i]
        intval = val / (1 + interest) ** (i + 1 - age)
        initval = initval + intval
        
    return initval

def getnpx2(mt, age, interest):
    val = mt[age]
    initval = 1
    for i in range(age + 2, 55):
        val *= mt[i - 1]
        if mt[age]==0:
            intval =0
        else:
            intval = val / (1 + interest) ** (i - age - 1) / mt[age]
        initval = initval + intval
    return initval

def getnpxtocert(mt, age, maxvalue):
    val = mt[age]
    for i in range(age + 1, min(maxvalue, 7)):
        val = val * mt[i]
    return val

And 1 large function which call all the small functions:

def calcannfactprelim(pval, age, intrate, certper):
    npx = getnpx(pval, age + int(certper), intrate)
    npx2 = getnpx2(pval, age + int(certper), intrate)

    if certper == 0:
        index = 1
        index2 = pval[age + int(certper)]

    else:
        index = getnpxtocert(pval, age,
                             age + int(certper)) 
        index2 = getnpxtocert(pval, age,age + int(certper) + 1)

    return index*npx+index2*npx2

These are the variables to use:

pval = np.array([0.000291,0.00027,0.000257,0.000294,0.000325,0.00035,0.000371,0.000388,0.000402,0.000414,0.000425,0.000437,0.011016,0.012251,0.013657,0.015233,0.016979,0.018891,0.020967,0.023209,0.025644,0.028304,0.03122,0.034425,0.037948,0.041812,0.046037,0.050643,0.055651,0.06108,0.066948,0.073275,0.080076,0.08737,0.095169,0.103455,0.112208,0.121402,0.131017,0.14103,0.151422,0.162179,0.173279,0.184706,0.196946,0.210484,0.225806,0.243398,0.263745,0.287334,0.314649,0.346177,0.382403,0.423813,0.470893])
age=3
intrate=0.04
certper=1

Regular function test result:

start=time.time()
print(calcannfactprelim(pval, age, intrate, certper))
print(time.time()-start)

Output is:

0.0002941874880982305 #result
0.0                   #time

In order to make this function faster ,I'm using python multiple processing to parallelly run it.

import multiprocessing

def calcannfactprelim_v(pval, age, intrate, certper):
    
    p1 = multiprocessing.Process(target=getnpx, args=(pval, age, intrate, certper,))
    p2 = multiprocessing.Process(target=getnpx2, args=(pval, age, intrate, certper,))

    # starting process 1
    p1.start()
    # starting process 2
    p2.start()

    # wait until process 1 is finished
    p1.join()
    # wait until process 2 is finished
    p2.join()

    # both processes finished
    if certper == 0:
        index = 1
        index2 = pval[age + int(certper)]

    else:
        index = getnpxtocert(pval, age,
                             age + int(certper)) 
        index2 = getnpxtocert(pval, age,age + int(certper) + 1)


    return index*npx+index2*npx2

But I have no idea how to return the value in this situation, any friends can help to complete the code and give a test?

With multiprocessing.Process() you'd need to create a queue for the processes to return values back to the main process. Alternatively, you can use multiprocess pools, which will do the same thing for you. There's some simple examples here — sj95126
– sj95126, Commented Oct 7, 2021 at 22:21

tdelaney · Accepted Answer · 2021-10-07 23:48:06Z

You need some channel to return the data. This is done for you in a multiprocessing pool.

def calcannfactprelim_v(pval, age, intrate, certper):
    with multiprocessing.pool(2) as pool:
        a1 = pool.apply_async(getnpx, (pval, age, intrate, certper,))
        a2 = pool.apply_async(getnpx2, (pval, age, intrate, certper,))
        result1 = a1.get()
        result2 = a2.get()

    # both processes finished
    if certper == 0:
        index = 1
        index2 = pval[age + int(certper)]

    else:
        index = getnpxtocert(pval, age,
                             age + int(certper)) 
        index2 = getnpxtocert(pval, age,age + int(certper) + 1)
    return index*npx+index2*npx2

You could even create a pool of 1 and do one of the calls in the main program. Suppose one returned a large value and the other a small one. Do the large one in your main so that the overhead of getting the value from the process back to the parent is less.

Kiril · Accepted Answer · 2021-10-07 22:34:08Z

Create an object in the parent process, say,

class ResultHolder():
     def __init__():
          result_a = None
          ………
          result_n = None

Pass this class as an argument to the functions which are to be run in the child processes and let the functions write the result in, say:

instance_of_result_holder.result_a = the_function_output

I believe you can do that by passing a dictionary or a list too. Long story short, objects are passed in a way akin to pass-by-reference, which means you can use them as intermediaries between the two separate worlds (contexts) of the parent and child processes (i.e. a child process can mutate an object defined in the parent one of you pass the latter as an argument to the respective function).

Pools and queues are another way to achieve that.

Collectives™ on Stack Overflow

Python how to use multiprocessing to different functions with return

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related