Why does Celery work in Python shell, but not in my Django views? (import problem)

Question

I installed Celery (latest stable version.) I have a directory called /home/myuser/fable/jobs. Inside this directory, I have a file called tasks.py:

from celery.decorators import task
from celery.task import Task

class Submitter(Task):
    def run(self, post, **kwargs):
        return "Yes, it works!!!!!!"

Inside this directory, I also have a file called celeryconfig.py:

BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "abc"
BROKER_PASSWORD = "xyz"
BROKER_VHOST = "fablemq"

CELERY_RESULT_BACKEND = "amqp"
CELERY_IMPORTS = ("tasks", )

In my /etc/profile, I have these set as my PYTHONPATH:

PYTHONPATH=/home/myuser/fable:/home/myuser/fable/jobs

So I run my Celery worker using the console ($ celeryd --loglevel=INFO), and I try it out. I open the Python console and import the tasks. Then, I run the Submitter.

>>> import fable.jobs.tasks as tasks
>>> s = tasks.Submitter()
>>> s.delay("abc")
<AsyncResult: d70d9732-fb07-4cca-82be-d7912124a987>

Everything works, as you can see in my console

[2011-01-09 17:30:05,766: INFO/MainProcess] Task tasks.Submitter[d70d9732-fb07-4cca-82be-d7912124a987] succeeded in 0.0398268699646s:

But when I go into my Django's views.py and run the exact 3 lines of code as above, I get this:

[2011-01-09 17:25:20,298: ERROR/MainProcess] Unknown task ignored: "Task of kind 'fable.jobs.tasks.Submitter' is not registered, please make sure it's imported.": {'retries': 0, 'task': 'fable.jobs.tasks.Submitter', 'args': ('abc',), 'expires': None, 'eta': None, 'kwargs': {}, 'id': 'eb5c65b4-f352-45c6-96f1-05d3a5329d53'}
Traceback (most recent call last):
  File "/home/myuser/mysite-env/lib/python2.6/site-packages/celery/worker/listener.py", line 321, in receive_message
    eventer=self.event_dispatcher)
  File "/home/myuser/mysite-env/lib/python2.6/site-packages/celery/worker/job.py", line 299, in from_message
    eta=eta, expires=expires)
  File "/home/myuser/mysite-env/lib/python2.6/site-packages/celery/worker/job.py", line 243, in __init__
    self.task = tasks[self.task_name]
  File "/home/myuser/mysite-env/lib/python2.6/site-packages/celery/registry.py", line 63, in __getitem__
    raise self.NotRegistered(str(exc))
NotRegistered: "Task of kind 'fable.jobs.tasks.Submitter' is not registered, please make sure it's imported."

It's weird, because the celeryd client does show that it's registered, when I launch it.

[2011-01-09 17:38:27,446: WARNING/MainProcess]  
Configuration ->
    . broker -> amqp://GOGOme@localhost:5672/fablemq
    . queues ->
        . celery -> exchange:celery (direct) binding:celery
    . concurrency -> 1
    . loader -> celery.loaders.default.Loader
    . logfile -> [stderr]@INFO
    . events -> OFF
    . beat -> OFF
    . tasks ->
        . tasks.Decayer
        . tasks.Submitter

Can someone help?

Are you also using django-celery or just celery by itself?

Thierry Lam
– Thierry Lam

2012-03-20 18:37:47 +00:00
Commented Mar 20, 2012 at 18:37 — Thierry Lam
– Thierry Lam, Commented Mar 20, 2012 at 18:37

2 revs · Accepted Answer · 2011-08-01 14:45:30Z

11

This is what I did which finally worked

in Settings.py I added

CELERY_IMPORTS = ("myapp.jobs", )

under myapp folder I created a file called jobs.py

from celery.decorators import task

@task(name="jobs.add")
def add(x, y):
    return x * y

Then ran from commandline: python manage.py celeryd -l info

in another shell i ran python manage.py shell, then

>>> from myapp.jobs import add
>>> result = add.delay(4, 4)
>>> result.result

and the i get:

16

The important point is that you have to rerun both command shells when you add a new function. You have to register the name both on the client and and on the server.

:-)

edited Aug 1, 2011 at 14:45

community wiki

2 revs
David Dehghan

Sign up to request clarification or add additional context in comments.

4 Comments

sequoia Over a year ago

Thanks for this solution, it is so simple, but it was stopping the show on my project. They should include the CELERY_IMPORTS line into the tutorial.

mattdeboard Over a year ago

Thanks for posting your solution. I was actually explicitly told that CELERY_IMPORTS isn't needed with django-celery, that djcelery.setup_loader() handled that, by a contributor in #celery. Disappointing that I had to chase a wild goose needlessly.

citadelgrad Over a year ago

-1 The documentation say explicitly not to import it as "myapp.jobs". I know that in this example you are specifying the name but this example is much more confusing than doing it the right way. docs.celeryq.org/en/latest/userguide/…

ApPeL Over a year ago

I had the same problem and by simple adding the CELERY_IMPORTS to my app it worked, thanks.

Seth · Accepted Answer · 2011-01-10 01:46:56Z

10

I believe your tasks.py file needs to be in a django app (that's registered in settings.py) in order to be imported. Alternatively, you might try importing the tasks from an __init__.py file in your main project or one of the apps.

Also try starting celeryd from manage.py:

$ python manage.py celeryd -E -B -lDEBUG

(-E and -B may or may not be necessary, but that's what I use).

answered Jan 10, 2011 at 1:46

Seth

46.8k10 gold badges87 silver badges123 bronze badges

5 Comments

TIMEX Over a year ago

What do you mean "needs to be in a django app"? I have tasks.py as a separate file, and import it in my views.py (import fable.jobs.tasks as tasks)

Seth Over a year ago

The tasks have to be registered in order to be seen by celery. That won't happen automatically in a views.py file, since that file isn't automatically loaded by python. I'm pretty sure that Celery will auto-find tasks in a file called tasks.py that's in a django app, or you can try importing the tasks in an __init__.py.

TIMEX Over a year ago

How come when I run the python shell (anywhere), it can detect and register the tasks?

TIMEX Over a year ago

I think I solved it. I put: sys.path.append("../jobs") to my views.py, and it worked!

Lionel Over a year ago

Also, I had to run celeryd in the directory that contained celeryconfig.py

asksol · Accepted Answer · 2011-01-10 09:51:54Z

5

See Automatic Naming and Relative Imports, in the docs:

http://celeryq.org/docs/userguide/tasks.html#automatic-naming-and-relative-imports

The tasks name is "tasks.Submitter" (as listed in the celeryd output), but you import the task as "fable.jobs.tasks.Submitter"

I guess the best solution here is if the worker also sees it as "fable.jobs.tasks.Submitter", it makes more sense from an app perspective.

CELERY_IMPORTS = ("fable.jobs.tasks", )

answered Jan 10, 2011 at 9:51

asksol

19.6k5 gold badges67 silver badges69 bronze badges

Collectives™ on Stack Overflow

Why does Celery work in Python shell, but not in my Django views? (import problem)

3 Answers 3

4 Comments

5 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related