Python Django Celery doesn't pick the task from the RabbitMQ

Question

I am working in a Python Django project.

One of its functionality is when someone send some data from a web application, foreward them to a mobile device.

For this functionality, application uses third party services as GCM & Amazon SNS. Following is the data flow of the application.

Web Application ------> Amazon SNS -------> GCM -------> Android Device

Due to the possible network delays, operational delays and failures in Amazon SNS side, web application uses celery as the scheduler application to achieve behaviors like asynchronous, retrying etc. Celery stores SNS sending task in a queue and process them asynchronously.

Celery as the scheduler uses RabbitMQ as its broker application. All together following is the data flow.

Web App --> Celery --> RabbitMQ --> Amazon SNS --> GCM --> Android Device

An issue is identified related to the Celery and RabbitMQ components.

We can see when celery sends a task to RabbitMQ, particular RabbitMQ queue count grows which means celery successfully created a task object and sent to the queue.

But particular task is exist in the queue forever(queue count doesn't reduced) and doesn't proceed to send the message to the Amazon SNS, which means Celery doesn't pick the stored task again from the queue.

(I could successfully push a message to the SNS bypassing the Celery which means there is no connection issue to the SNS.)

Which may cause celery not to pick the stored task from the RabbitMQ ?

Wu Wenter · Accepted Answer · 2018-04-17 07:49:47Z

1

You may figure out which is task producer and which is task consumer. Celery works as task consumer, I think the web app is the task producer.

The data flow seems like "WebApp -> RabbitMQ(Broker) -> Celery" .

So, make sure the celery broker configured properly, and the celery worker started successfully.

More info please visit: Celery Broker Configuration

And if you want to use celery as periodic task consumer. You may visit celery periodic task

edited Apr 17, 2018 at 7:49

answered Apr 11, 2018 at 5:24

Wu Wenter

1715 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Udara Seneviratne Over a year ago

What do you mean by celery broker in "So, make sure the celery broker configured properly." statement ? You mean RabbitMQ ?

Wu Wenter Over a year ago

Yeah, Celery broker support several backends like redis, rabbitmq,mysql .. The broker will save the serialized tasks in queue . The producer push task to broker, and the consumer read tasks from broker.

Udara Seneviratne Over a year ago

With the help of the points you provided, I could solve the issue. I was not aware of celery worker thread to start to process the messages in the queue. So, none of celery workers is started in my setup. So, issue is not acctually related to celery broker configuration but none of celery worker was started. Could you please edit your answer to emphasize that point. If so, I can mark your answer as correct.

Collectives™ on Stack Overflow

Python Django Celery doesn't pick the task from the RabbitMQ

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related