How to seamlessly maintain code of django celery in a multi node environment

Question

I have a Django application which uses django-celery, celery and rabbitmq for offline, distributed processing.

Now the setup is such that I need to run the celery tasks (and in turn celery workers) in other nodes in the network (different from where the Django web app is hosted).

To do that, as I understand I will need to place all my Django code in these separate servers. Not only that, I will have to install all the other python libraries which the Django apps require.

This way I will have to transfer all the django source code to all possible servers in the network, install dependencies and run some kind of an update system which will sync all the sources across nodes.

Is this the right way of doing things? Is there a simpler way of making the celery workers run outside the web application server where the Django code is hosted ?
If indeed there is no way other than to copy code and replicate in all servers, is there a way to copy only the source files which the celery task needs (which will include all models and views - not so small a task either)

sean · Accepted Answer · 2013-06-24 07:39:16Z

2

For this type of situation I have in the past made a egg of all of my celery task code that I can simply rsync or copy in some fashion to my worker nodes. This way you can edit your celery code in a single project that can be used in your django and on your work nodes.

So in summary create a web-app-celery-tasks project and make it into an installable egg and have a web-app package that depends on the celery tasks egg.

answered Jun 24, 2013 at 7:39

sean

3,98523 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

binithb Over a year ago

do you mean I should create an egg file with all the model classes that I need ? I guess I will need to pack the settings.py file as well, since the task will be using the model classes to talk to the database and the database configuration details are mentioned only in the settings.py file

sean Over a year ago

Ya, so it seems like you will have three egg's the web front-end, celery tasks, and the database models. Doing it this way you can update any single part of these chunks without much worry of breaking other projects or deploy unnecessarily.

Collectives™ on Stack Overflow

How to seamlessly maintain code of django celery in a multi node environment

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related