Django celery tasks in separate server

Question

We have two servers, Server A and Server B. Server A is dedicated for running django web app. Due to large number of data we decided to run the celery tasks in server B. Server A and B uses a common database. Tasks are initiated after post save in models from Server A,webapp. How to implement this idea using rabbitmq in my django project

That is exactly how Celery is supposed to work, and there is nothing unusual here. Where are you having problems? — Daniel Roseman
– Daniel Roseman, Commented May 22, 2017 at 21:55

rowman · Accepted Answer · 2020-08-27 14:08:51Z

27

You have 2 servers, 1 project and 2 settings(1 per server).

server A (web server + rabbit)

server B (only celery for workers)

Then you set up the broker url in both settings. Something like this:

BROKER_URL = 'amqp://user:password@IP_SERVER_A:5672//' matching server A to IP of server A in server B settings.

For now, any task must be sent to rabbit in server A to virtual server /.

In server B, you must just initialize celery worker, something like this:

python manage.py celery worker -Q queue_name -l info

and thats it.

Explanation: django sends messages to rabbit to queue a task, then celery workers request some message to execute a task.

Note: Is not required that rabbitMQ have to be installed in server A, you can install rabbit in server C and reference it in the BROKER_URL in both settings(A and B) like this: BROKER_URL='amqp://user:password@IP_SERVER_C:5672//'.

Sorry for my English.

greetings.

edited Aug 27, 2020 at 14:08

rowman

1,6262 gold badges18 silver badges27 bronze badges

answered May 22, 2017 at 14:04

Diego Puente

2,0341 gold badge23 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Jenish Over a year ago

Nothing wrong in your English. Crisp and clear information. Thanks

tech4242 Over a year ago

@Diego how would you theoretically point to a second worker D on a different machine and let RabbitMQ (assuming it's the broker) handle the load from Celery? Is it done automatically as long as you list all workers?

Diego Puente Over a year ago

@tech4242 I don't understand, what do you mean with "worker D"? you want many workers by server? can you reformulate? This question does not mention about AMPQ protocol(or how rabbit does put a message on a queue).

Diego Puente Over a year ago

@tech4242 the broker don't need to know the number of workers. The broker just have to know where put the message(in what queue). If you want more workers, instructions for server B will work for server N. In each settings you set broker url and in the start command you tell from what queue you want to process mesaages.

Diego Puente Over a year ago

obviously you can add many workers by server with the same command.

|

Collectives™ on Stack Overflow

Django celery tasks in separate server

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related