How to use multiprocessing.process() in AWS Lambda to replace .pool()?

Question

So, after doing some homework on this problem it seems that multiprocessing.pool() won't work in an AWS Lambda, but multiprocessing.process() should work. The problem is that when I try to swap out the code that I know works with the new .process() code, I get a list of instead of the values I'm looking for. Here's what I'm trying to do:

(note: the commented code is the .pool() example that works, whereas the .process() right below it is what breaks)

def evaluate_plans_parallel(foo, baz, bar):
score = {}
input_tuple = []
for k, v in plans.items():
    input_tuple.append({'somevar1': k, 'somevar3': v, 'somevar': foo,'somevar2': baz})

# score_list = []

# pool = multiprocessing.Pool(get_parallel_count())
# pool = multiprocessing.Pool(3)
# score_list = pool.map(evaluate_single_plan, input_tuple)

processes = []

score_list = []
for tup in input_tuple:
    process = Process(target=evaluate_single_plan, args=(tup,))
    processes.append(process)


for process in processes:
    process.start()

for process in processes:
    process.join()

print(processes)

for x in processes:
    for k, v in x.items():
        score[k] = v

top_score = sorted(score.values())[0]
print('Top score: {}: '.format(round(top_score, 3)))
return score

Am I doing something obviously incorrect?

yoursweater · Accepted Answer · 2019-02-14 19:35:09Z

1

So it turns out I needed to sync up the results using .pipe():

processes = []
parent_connections = []

for tup in input_tuple:
    parent_conn, child_conn = Pipe()
    parent_connections.append(parent_conn)

    process = Process(target=evaluate_single_plan, args=(tup, child_conn,))
    processes.append(process)


for process in processes:
    process.start()

for process in processes:
    process.join()

for x in parent_connections:
    print(x.recv()])

and in the evaluate_single_plan function, instead of returning a value I just needed to send to the child_conn argument that was passed in:

child_conn.send(result)
child_conn.close()

answered Feb 14, 2019 at 19:35

yoursweater

2,0773 gold badges27 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to use multiprocessing.process() in AWS Lambda to replace .pool()?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related