Executing two tasks at the same time with Celery

Question

I'm testing celery in a local environment. My Python file has the following two lines of code:

celery_app.send_task('tasks.test1', args=[self.id], kwargs={})
celery_app.send_task('tasks.test2', args=[self.id], kwargs={})

Looking at the console output they seem to execute one after another in sequence. But test2 only runs after test1 has finished. At least this is the way it seems reading the console output.

These tasks have no dependancies on each other so I don't want one task waiting for another to complete before moving onto the next line.

How can I execute both tasks as the same time?

---- **** -----
--- * ***  * -- Darwin-14.0.0-x86_64-i386-64bit
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         tasks:0x104cd8c10
- ** ---------- .> transport:   sqs://123
- ** ---------- .> results:     disabled
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ----
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery

@ndpu so its a concurrency issue? I don't have to use parallel tasks here? — Prometheus
– Prometheus, Commented Aug 29, 2014 at 15:07
it states on start up that I have "concurrency: 4 (prefork)" — Prometheus
– Prometheus, Commented Aug 29, 2014 at 15:10

Chillar Anand · Accepted Answer · 2020-02-27 13:18:51Z

28

There are multiple ways to achieve this.

1. Single Worker - Single Queue.

$ celery -A my_app worker -l info  -c 2 -n my_worker

This will start a worker which executes 2 tasks at the same time.

2. Multiple workers - Single Queue.

$ celery -A my_app worker -l info  -c 1 -n my_worker1
$ celery -A my_app worker -l info  -c 1 -n my_worker2

This will start two workers which executes one task at a time. Note both tasks are in the same queue.

3. Multiple workers - Multiple Queues.

$ celery -A my_app worker -l info  -c 1 -n my_worker1 -Q queue1
$ celery -A my_app worker -l info  -c 1 -n my_worker2 -Q queue2

This will start two workers which executes one task at a time. But here you have route the tasks accordingly.

celery_app.send_task('tasks.test1', args=[self.id], kwargs={}, queue='queue1')
celery_app.send_task('tasks.test2', args=[self.id], kwargs={}, queue='queue2')

4. Single worker - All Queues

$ celery -A my_app worker -l info -n my_worker1

If you don't mention any queue, it will consume from all queues by default.

edited Feb 27, 2020 at 13:18

answered Aug 30, 2014 at 6:04

Chillar Anand

29.8k10 gold badges126 silver badges144 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Chillar Anand Over a year ago

@user1012513 You can run celery -A my_app worker -l info -c 1 -n my_worker1 -Q queue1,queue2,queue3

Aditya Nagesh Over a year ago

If you do not mention have -Q flag, the workers by default reads from all the queues.

Chillar Anand Over a year ago

@AdityaNagesh Yes, you are right. Updated in answer.

sattva_venu Over a year ago

@ChillarAnand, in my case even when I use -c 2 in celery worker command, one task is blocking another task. The command I am using is celery worker -A celery_app -P gevent -l info -c 2. Can you let me know what else should I change so that both tasks can be run in parallel?

Guilherme Over a year ago

@sattva_venu did you find a solution? I'm having the same issue

All Іѕ Vаиітy · Accepted Answer · 2017-10-27 17:42:36Z

7

Call the worker with --autoscale option which would scale up and down processes as required.

--autoscale AUTOSCALE
                       Enable autoscaling by providing max_concurrency,
                       min_concurrency. Example:: --autoscale=10,3 (always
                       keep 3 processes, but grow to 10 if necessary)

example.

celery -A sandbox worker --autoscale=10,0 --loglevel=info

answered Oct 27, 2017 at 17:42

All Іѕ Vаиітy

26.7k16 gold badges97 silver badges114 bronze badges

2 Comments

sattva_venu Over a year ago

Does this work on windows? I gave a try but still one celery task is blocking another celery task.

All Іѕ Vаиітy Over a year ago

@sattva_venu Sorry I don't use Windows, no Idea.

Collectives™ on Stack Overflow

Executing two tasks at the same time with Celery

2 Answers 2

5 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related