Assign a specific number of workers for each Celery task in Python

Question

I have three Celery tasks:

prediction
training
healthcheck

And I have 4 cpus in the machine. I would like to allocate 3 workers for prediction and training, and 1 for healthcheck. What is the easiest way to implement that? Note that I am going to schedule these tasks, and thus, I can't specify the number of workers in the apply_async() function directly.

Actual configs:

CELERY = Celery(
    CELERY_APP_NAME,
    backend=CELERY_BACKEND,
    broker=CELERY_BROKER,
    include=["src.tasks"],
)

CELERY.conf.update({"task_routes": {"src.tasks.*": {"queue": "input_queue"}},
                    }
                   )


@CELERY.task
def prediction():
    pass

@CELERY.task
def training():
    pass

@CELERY.task
def healthcheck():
    pass

And the command to run the worker:

celery --loglevel=INFO -A src.tasks worker -Q input_queue

DejanLekic · Accepted Answer · 2020-09-01 15:35:25Z

1

This is how I would do it, because I could never truly understand celery multi:

celery -A yourproject.yourapp -l info -c 3 -Q prediction
celery -A yourproject.yourapp -l info -c 1 -Q healthcheck

After you run something similar to the above (with correct application parameter) you will end up with two workers subscribed to different queues. Your prediction and training tasks would be triggered with queue=prediction among other named parameters, while healthcheck task should be sent to the healthcheck queue in the similar way.

answered Sep 1, 2020 at 15:35

DejanLekic

20k4 gold badges53 silver badges84 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

nolwww Over a year ago

Thanks, thats exactly how I did it finally.

Collectives™ on Stack Overflow

Assign a specific number of workers for each Celery task in Python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related