Django celery run multiple workers with different queues

Question

i try to configure three queues/workers for celery in django.

settings.py

CELERY_BROKER_URL = 'redis://localhost:6379'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Europe/Berlin'
CELERY_QUEUES = (
    Queue('manually_task', Exchange('manually_task'), routing_key='manually_task'),
    Queue('periodically_task', Exchange('periodically_task'), routing_key='periodically_task'),
    Queue('firsttime_task', Exchange('firsttime_task'), routing_key='firsttime_task'),
)
CELERY_ROUTES = {
    'api.tasks.manually_task': {
        'queue': 'manually_task',
        'routing_key': 'manually_task',
    },
    'api.tasks.periodically_task': {
        'queue': 'periodically_task',
        'routing_key': 'periodically_task',
    },
    'api.tasks.firsttime_task': {
        'queue': 'firsttime_task',
        'routing_key': 'firsttime_task',
    },
}

I have three tasks and every task should be have their own queue/worker. My tasks look like this:

@shared_task
def manually_task(website_id):
    print("manually_task");
    website = Website.objects.get(pk=website_id)
    x = Proxy(website, "49152")
    x.startproxy()
    x = None


@periodic_task(run_every=(crontab(hour=19, minute=15)), ignore_result=True)
def periodically_task():
    websites = Website.objects.all()

    for website in websites:
        x = Proxy(website, "49153")
        x.startproxy()
        x = None


@shared_task
def firsttime_task(website_id):
    website = Website.objects.get(pk=website_id)
    x = Proxy(website, "49154")
    x.startproxy()
    x = None

Now for the first trial i start only one worker:

celery -A django-proj worker -Q manually_task -n manually_task

My problem is that the task not execute apparently, "manually_task" not printed. Why its not working?

how did you invoke your tasks after the worker has been started? — Greenev
– Greenev, Commented Sep 30, 2019 at 8:32
In a view for example like this: manually_task.delay(webseite.pk) — Basti G.
– Basti G., Commented Sep 30, 2019 at 10:06
With one worker and without CELERY_QUEUES and CELERY_ROUTES settings in the settings.py it works fine. — Basti G.
– Basti G., Commented Sep 30, 2019 at 10:07

Greenev · Accepted Answer · 2019-10-03 08:02:23Z

1

Based on the commments I suggest you should either provide queue name when you are calling a task from a view like manually_task.apply_async((webseite.pk,), queue='manually_task'), you or add default queue named celery when you start the worker as in celery -A django-proj worker -Q manually_task,celery

answered Oct 3, 2019 at 8:02

Greenev

9096 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Django celery run multiple workers with different queues

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related