Parallel and Sequential Execution of tasks using Celery

Question

I am pretty new to Celery and planning to use it for scheduling few jobs. One of the scenario is as below.

TASK_NM | DEPENDENCY

T1

T2      |        T1

T3      |        T1 

T4

T5      |        T3 and T4

From the above case,

I have to execute T1 and T4 in parallel
After Successful completion of T1, execute T2 and T3 in parallel
After Successful completion of T3 and T4, execute T5.

If any task failed in the middle, do not execute the dependent jobs.

How can we achieve this using Celery?

Thanks in Advance!!!

2ps · Accepted Answer · 2022-07-23 20:45:01Z

5

Yes, but it will require some work. You will have to use chain, group, and chord to get the behavior that you want:

Execute T1, T2 and T3 serially by using chain:
```
res = chain(T1.si(), T2.si(), T3.si())
```
Execute (1) in parallel with T4 by using group:
```
res2 = group(res.s(), T4.s())
```
Execute T5 after T3 and T4 are complete by using chord:
```
res3 = chord(res2.s(), T5.s())
```
Execute it all
```
result = res3().get() 
```

The docs do a pretty good job of explaining which to use for serial and which to use for parallel. The tricky part to remember is that the chain, chord and group all return tasks themselves so have to be invoked using (), delay() or apply_async().

edited Jul 23, 2022 at 20:45

answered Jan 26, 2017 at 7:40

2ps

16k2 gold badges34 silver badges53 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Parallel and Sequential Execution of tasks using Celery

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related