Active Django settings file from Celery worker

Question

So I already looked around a lot for this but couldn't find a good answer. I'm using Celery 3.1.7 and Django 1.5.1., without django-celery package since newer versions of Celery don't require it anymore. I managed to set up tasks and execute them using RabbitMQ. Everything is working as it should there. However, I am integrating this in a existing, quite large, Django project. There we specified couple of Django settings files, not just one. We run different one depending on environment, for instance one for local machines and one for server. My problem is that I can't seem to be able to track down which settings file is "active" from the celery worker, which runs celery.py file in my project root (as documentation specifies). There the documentation requires to specify Django settings file like this:

os.environ.setdefault('DJANGO_SETTINGS_MODULE', "project.settings.server")

Now this works, but if I move the stuff locally I need to change it to settings.local to make it work, and that every time. Reading settings object in runtime like I do in standard Django files didn't work since celery worker executes in a different process. So, using this situation, does anyone have any idea on how to dynamically fetch active Django settings file from celery worker? Or perhaps pass it in as a variable when starting celery worker? (like for Django, etc --settings=project.settings.local) Thanks!

lkiss80 · Accepted Answer · 2014-09-29 18:03:32Z

43

When initializing the celery worker on the command line, just set the environment variable prior to the celery command.

DJANGO_SETTINGS_MODULE='proj.settings' celery -A proj worker -l info

answered Sep 29, 2014 at 18:03

lkiss80

7769 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Rajesh Yogeshwar Over a year ago

I know its too old a thread. But I tumbled upon this because I needed same thing. I did export DJANGO_SETTINGS_MODULE='proj.settings' celery -A proj worker -l info

Cody Over a year ago

how am I supposed to do this from supervisor ?

Florian Over a year ago

@Cody using "environment" in your [program:x] section supervisord.org/…

Ángel Jiménez Over a year ago

@RajeshYogeshwar comment worked as a charm for me! Great workaround for when the settings are separated by environments.

olofom · Accepted Answer · 2014-02-26 23:29:15Z

4

One idea could be to have Django load the correct settings file every time and let celery use the one Django is using. That's how I'm doing it.

Say you have the project structure:

project/
    proj/
        settings.py
        urls.py
        ...

Replace with

project/
    proj/
        settings/
            __init__.py
            local.py
            production.py
            common_settings.py
            ...
        urls.py
        ...

Let common_settings.py be all settings shared between all environments and let your init.py load whatever config should be used.

# __init__.py:

from common_settings import *

# A check for environment variable, hostname, etc:
# Example for checking hostname:
from platform import node
if node() in ['dev1', 'dev2']:
    from local import *
elif node() in ['prod1', 'prod2']:
    from production import *

Now you can always rely on project.proj.settings being the right settings file for your environment.

answered Feb 26, 2014 at 23:29

olofom

6,56112 gold badges39 silver badges51 bronze badges

2 Comments

MaRiNkO Over a year ago

I actually like this approach, looks like a good start! Many thanks!

olofom Over a year ago

@Nirri are you sure that you import all the other files * in your init.py in the settings folder? Doing that is the equivalent of having a plain settings file so that should definitely work no matter which version of django you're on.

Adrián · Accepted Answer · 2014-02-26 23:47:48Z

3

setdefault() returns the value of the variable in the system environment. It will return "project.settings.server" only if DJANGO_SETTINGS_MODULE is not defined.

So, I would leave the most frequently used settings module in there and change it when needed by explicitly declaring the environment variable:

i.e. for local development, in your virtualenv hooks, in your .bashrc, manually, etc:

export DJANGO_SETTINGS_MODULE=project.settings.local.

answered Feb 26, 2014 at 23:47

Adrián

6,2651 gold badge29 silver badges49 bronze badges

1 Comment

MaRiNkO Over a year ago

A good tip, but, I have multiple server settings as well for multiple server environments, so that would require setting up hooks on each of them which breaks my idea of "work out of the box". Otherwise a good approach, might use it for some other stuff, thanks!

bigjust · Accepted Answer · 2014-10-22 20:11:40Z

-1

found this buried in the celery docs:

# project/app/tasks.py
app = Celery()
app.config_from_object('django.conf:settings')

http://celery.readthedocs.org/en/latest/whatsnew-3.1.html#django-supported-out-of-the-box

This will simply pull celery settings from the django settings file, which is usually what you want for environment separation.

answered Oct 22, 2014 at 20:11

bigjust

2991 silver badge5 bronze badges

Collectives™ on Stack Overflow

Active Django settings file from Celery worker

4 Answers 4

4 Comments

2 Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

4 Comments

2 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related