Running celery group: where two tasks are run in parallel to get results to produce a third result

Question

How do I set it so that the dependent tasks are run in parallel (instead of sequentially) to reduce the overall execution time

I have the following "master" task below
It is dependent on the get_latest_close_price(symbol) and option_chain(symbol) celery task results
The master tasks transforms the results from these two tasks to produce the final pivot table
The problem is that both these independent tasks talk to different API systems to get it's input (sometimes taking several seconds to execute)
What I am noticing is that the order of the celery task in the group statement matters:
- If get_lastest_close_price is first, it gets run before the option_chain (and vise versa)
- To me it seems like these statements are being run sequentially instead of in parallel
- Is my understanding wrong?

.

@celery.task(name='master_task')
def process_chain(symbol):

   # g = group(get_latest_close_price.s(symbol), option_chain.s(symbol))
   g = group(option_chain.s(symbol), get_latest_close_price.s(symbol))

   results = g()

   with result.allow_join_result():
       data = results.get()
       data = util_transform_option_chain(data[1], data[0])

   return({'result':data})

2ps · Accepted Answer · 2019-11-19 03:53:52Z

1

group will definitely run in parallel . . . if you have multiple workers or concurrency > 1. But, just as an FYI--you may want to combine a chord with group to handle the result the way that you want.

answered Nov 19, 2019 at 3:53

2ps

16k2 gold badges34 silver badges53 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Karun Over a year ago

setting --concurrency=5 got me the result I was expecting

Karun Over a year ago

Is there a good tutorial/article to understand group, chord etc. I'm not sure if I understand the concepts fully...I'm sort of wingin' it

Collectives™ on Stack Overflow

Running celery group: where two tasks are run in parallel to get results to produce a third result

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related