Django celery tasks number limit issue

Question

I have problem run celery workers to execute same task multiple times in parallel.

I ran 3 workers and set --concurrency to 2 for all workers.

But it only executes 3 tasks over all 3 workers.

I hope to run about 10 workers.

celery -A my_app worker -l info  -c 2 -n worker1
celery -A my_app worker -l info  -c 2 -n worker2
celery -A my_app worker -l info  -c 2 -n worker3

Please help me whether I can run more than 3 tasks at a time.

You can check autoscale option in celery.

Manoj Jadhav
– Manoj Jadhav

2017-11-27 04:58:38 +00:00
Commented Nov 27, 2017 at 4:58 — Manoj Jadhav
– Manoj Jadhav, Commented Nov 27, 2017 at 4:58

Nikita Kozel · Accepted Answer · 2017-11-26 23:19:54Z

1

I solved it by using max-tasks-per-child argument.

Here is my solution.

celery -A my_app worker -l info -c 10 --max-tasks-per-child 10 -n worker1

After running this, it can execute 10 same tasks at a time in parallel.

Hope this is helpful.

answered Nov 26, 2017 at 23:19

Nikita Kozel

215 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

TimB Over a year ago

The --max-tasks-per-child setting has nothing to do with parallel execution of tasks. It specifies how many tasks a child process executes before it is killed and replaced with a new process. The option -c 10 is what specifies the number of child processes, and hence the number of parallel tasks that can run.

Collectives™ on Stack Overflow

Django celery tasks number limit issue

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related