task with number of task can run along that in celery

Question

As I see in celery, It can get number of tasks for a worker, that can run them at a same time.

I need run a task and set number of tasks can run simultaneously with this task.

celery with workers

Therefore, If I set this number to 2 and this task send to worker with 10 threads, worker can run just one another task.

Olexander Yermakov · Accepted Answer · 2015-07-15 11:26:49Z

1

Worker will reserve tasks for each worker's tread. If you want to limit the number of tasks worker can execute the same time, you should configure your concurrency (e.g. to limit 1 task at the time, you need worker with 1 process -c 1).

You can also check prefetch configuration, but it only defines the number of tasks reserved for each process of the worker.

Here is Celery documentation where prefetch configuration explained: http://celery.readthedocs.org/en/latest/userguide/optimizing.html

answered Jul 15, 2015 at 11:26

Olexander Yermakov

3973 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

MKM Over a year ago

Is there any way to dynamic change number of worker process?

Olexander Yermakov Over a year ago

You can start/stop additional workers any time you want. Also it might be possible to change worker's pool to increase/decrease the number of processes for the particular worker, but I can answer how it's better to do. You can refer to this question, maybe it will be helpful: stackoverflow.com/questions/6956272/… Also Celery documentation on workers: celery.readthedocs.org/en/latest/userguide/workers.html

Collectives™ on Stack Overflow

task with number of task can run along that in celery

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related