2

Try to get it working: Django 1.6.11 + Celery 3.1.19 + Kombu 3.0.33 on Ubuntu 14.04 using tutorial http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html so i completely repeated the project structure from here: https://github.com/celery/celery/tree/3.1/examples/django/

When i run command from django project root: /var/www/engine$ celery worker -l INFO -B

Inside my app vk_wall i have tasks.py file with add task.

i get an error:

    graph@engine:/var/www/engine$ celery worker -l INFO -B
[2016-01-17 18:41:41,159: WARNING/MainProcess] /usr/local/lib/python3.4/dist-packages/celery/apps/worker.py:161: CDeprecationWarning:
Starting from version 3.2 Celery will refuse to accept pickle by default.

The pickle serializer is a security concern as it may give attackers
the ability to execute any command.  It's important to secure
your broker from unauthorized access when using pickle, so we think
that enabling pickle should require a deliberate action and not be
the default choice.

If you depend on pickle then you should set a setting to disable this
warning and to be sure that everything will continue working
when you upgrade to Celery 3.2::

    CELERY_ACCEPT_CONTENT = ['pickle', 'json', 'msgpack', 'yaml']

You must only enable the serializers that you will actually use.


  warnings.warn(CDeprecationWarning(W_PICKLE_DEPRECATED))

 -------------- celery@engine v3.1.19 (Cipater)
---- **** -----
--- * ***  * -- Linux-3.13.0-24-generic-x86_64-with-Ubuntu-14.04-trusty
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         default:0x7f9c6495cc18 (.default.Loader)
- ** ---------- .> transport:   amqp://guest:**@localhost:5672//
- ** ---------- .> results:     disabled
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ----
--- ***** ----- [queues]
 -------------- .> celery           exchange=celery(direct) key=celery


[tasks]


[2016-01-17 18:41:41,187: INFO/Beat] beat: Starting...
[2016-01-17 18:41:41,212: ERROR/Beat] Removing corrupted schedule file 'celerybeat-schedule': error(13, 'Permission denied')
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/kombu/utils/__init__.py", line 323, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'scheduler'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/celery/beat.py", line 376, in setup_schedule
    self._store = self._open_schedule()
  File "/usr/local/lib/python3.4/dist-packages/celery/beat.py", line 366, in _open_schedule
    return self.persistence.open(self.schedule_filename, writeback=True)
  File "/usr/lib/python3.4/shelve.py", line 239, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "/usr/lib/python3.4/shelve.py", line 223, in __init__
    Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
  File "/usr/lib/python3.4/dbm/__init__.py", line 94, in open
    return mod.open(file, flag, mode)
_gdbm.error: [Errno 13] Permission denied
[2016-01-17 18:41:41,219: ERROR/Beat] Process Beat
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/kombu/utils/__init__.py", line 323, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'scheduler'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/celery/beat.py", line 376, in setup_schedule
    self._store = self._open_schedule()
  File "/usr/local/lib/python3.4/dist-packages/celery/beat.py", line 366, in _open_schedule
    return self.persistence.open(self.schedule_filename, writeback=True)
  File "/usr/lib/python3.4/shelve.py", line 239, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "/usr/lib/python3.4/shelve.py", line 223, in __init__
    Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
  File "/usr/lib/python3.4/dbm/__init__.py", line 94, in open
    return mod.open(file, flag, mode)
_gdbm.error: [Errno 13] Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/billiard/process.py", line 292, in _bootstrap
    self.run()
  File "/usr/local/lib/python3.4/dist-packages/celery/beat.py", line 545, in run
    self.service.start(embedded_process=True)
  File "/usr/local/lib/python3.4/dist-packages/celery/beat.py", line 464, in start
    humanize_seconds(self.scheduler.max_interval))
  File "/usr/local/lib/python3.4/dist-packages/kombu/utils/__init__.py", line 325, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/usr/local/lib/python3.4/dist-packages/celery/beat.py", line 504, in scheduler
    return self.get_scheduler()
  File "/usr/local/lib/python3.4/dist-packages/celery/beat.py", line 499, in get_scheduler
    lazy=lazy)
  File "/usr/local/lib/python3.4/dist-packages/celery/utils/imports.py", line 53, in instantiate
    return symbol_by_name(name)(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/celery/beat.py", line 358, in __init__
    Scheduler.__init__(self, *args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/celery/beat.py", line 185, in __init__
    self.setup_schedule()
  File "/usr/local/lib/python3.4/dist-packages/celery/beat.py", line 378, in setup_schedule
    self._store = self._destroy_open_corrupted_schedule(exc)
  File "/usr/local/lib/python3.4/dist-packages/celery/beat.py", line 372, in _destroy_open_corrupted_schedule
    return self._open_schedule()
  File "/usr/local/lib/python3.4/dist-packages/celery/beat.py", line 366, in _open_schedule
    return self.persistence.open(self.schedule_filename, writeback=True)
  File "/usr/lib/python3.4/shelve.py", line 239, in open
    return DbfilenameShelf(filename, flag, protocol, writeback)
  File "/usr/lib/python3.4/shelve.py", line 223, in __init__
    Shelf.__init__(self, dbm.open(filename, flag), protocol, writeback)
  File "/usr/lib/python3.4/dbm/__init__.py", line 94, in open
    return mod.open(file, flag, mode)
_gdbm.error: [Errno 13] Permission denied[2016-01-17 18:41:41,227: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//

[2016-01-17 18:41:41,252: INFO/MainProcess] mingle: searching for neighbors
[2016-01-17 18:41:42,272: INFO/MainProcess] mingle: all alone
[2016-01-17 18:41:42,302: WARNING/MainProcess] celery@engine ready.
[2016-01-17 18:42:45,778: ERROR/MainProcess] Received unregistered task of type 'vk_wall.tasks.mul'.
The message has been ignored and discarded.

Did you remember to import the module containing this task?
Or maybe you are using relative imports?
Please see http://bit.ly/gLye1c for more information.

The full contents of the message body was:
{'expires': None, 'errbacks': None, 'callbacks': None, 'args': (2, 5), 'kwargs': {}, 'utc': True, 'retries': 0, 'eta': '2016-01-17T15:42:55.746877+00:00', 'task': 'vk_wall.tasks.mul', 'id': '38c3c3d3-3b53-4c58-a308-d41ed24ea613', 'timelimit': (None, None), 'chord': None, 'taskset': None} (302b)
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/celery/worker/consumer.py", line 455, in on_task_received
    strategies[name](message, body,
KeyError: 'vk_wall.tasks.mul'
[2016-01-17 18:42:45,786: ERROR/MainProcess] Received unregistered task of type 'vk_wall.tasks.add'.
The message has been ignored and discarded.

Did you remember to import the module containing this task?
Or maybe you are using relative imports?
Please see http://bit.ly/gLye1c for more information.

The full contents of the message body was:
{'expires': None, 'errbacks': None, 'callbacks': None, 'args': (2, 2), 'kwargs': {}, 'utc': True, 'retries': 0, 'eta': None, 'task': 'vk_wall.tasks.add', 'id': 'cc04ddae-024e-4884-9a10-7a7066206fa9', 'timelimit': (None, None), 'chord': None, 'taskset': None} (264b)
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/celery/worker/consumer.py", line 455, in on_task_received
    strategies[name](message, body,
KeyError: 'vk_wall.tasks.add'

I see celery beat issue, also in traceback i see celery can find add, mul, get_wallposts_by_owner_id tasks, but cannot execute them (in my views.py i cal them: from vk_wall.tasks import add, mul res_mul = mul.apply_async((2, 5), queue='celery', countdown=10) res_add = add.delay(2, 2))

I also installed django-celery to see tasks in admin panel.

By the way: i was able to get celery working and executing task during beginner tutorial http://docs.celeryproject.org/en/latest/getting-started/next-steps.html but now need django to access models in tasks.

UPD: my settings.py django file

    BROKER_URL = 'django://'

#: Only add pickle to this list if your broker is secured
#: from unwanted access (see userguide/security.html)
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_RESULT_EXPIRES=3600,
#CELERY_RESULT_BACKEND='amqp',
CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend',
#CELERY_RESULT_BACKEND='djcelery.backends.cache:CacheBackend',
CELERY_RESULT_SERIALIZER = 'pickle', #json
CELERY_TASK_SERIALIZER = 'pickle',
CELERY_IMPORTS=["vk_wall.tasks"]

UPD2: my celery.py file (located firstapp/firstapp, the same level as django's settings.py)

from __future__ import absolute_import

import os

from celery import Celery

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'firstapp.settings')

from django.conf import settings  # noqa

app = Celery('firstapp')

# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)


@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

UPD3: full traceback

    graph@engine:/var/www/engine$ celery worker -A firstapp.settings -l debug
    Traceback (most recent call last):
      File "/usr/local/bin/celery", line 11, in <module>
        sys.exit(main())
      File "/usr

/local/lib/python3.4/dist-packages/celery/__main__.py", line 30, in main
    main()
  File "/usr/local/lib/python3.4/dist-packages/celery/bin/celery.py", line 81, in main
    cmd.execute_from_commandline(argv)
  File "/usr/local/lib/python3.4/dist-packages/celery/bin/celery.py", line 770, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/usr/local/lib/python3.4/dist-packages/celery/bin/base.py", line 309, in execute_from_commandline
    argv = self.setup_app_from_commandline(argv)
  File "/usr/local/lib/python3.4/dist-packages/celery/bin/base.py", line 469, in setup_app_from_commandline
    self.app = self.find_app(app)
  File "/usr/local/lib/python3.4/dist-packages/celery/bin/base.py", line 489, in find_app
    return find_app(app, symbol_by_name=self.symbol_by_name)
  File "/usr/local/lib/python3.4/dist-packages/celery/app/utils.py", line 229, in find_app
    sym = symbol_by_name(app, imp=imp)
  File "/usr/local/lib/python3.4/dist-packages/celery/bin/base.py", line 492, in symbol_by_name
    return symbol_by_name(name, imp=imp)
  File "/usr/local/lib/python3.4/dist-packages/kombu/utils/__init__.py", line 96, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/celery/utils/imports.py", line 101, in import_from_cwd
    return imp(module, package=package)
  File "/usr/lib/python3.4/importlib/__init__.py", line 109, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 2254, in _gcd_import
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2224, in _find_and_load_unlocked
ImportError: No module named 'firstapp'
Improve this question
4
  • Do you have a @task or @shared_task decorators on your functions add, mul, etc? Commented Jan 17, 2016 at 16:40
  • 1
    Also check permissions on the file celerybeat-schedule wherever it is - your current user has no permissions to delete this file. It seems to be corrupted. Commented Jan 17, 2016 at 16:47
  • @baldr yes, tried decorators app.task and shared_task - the same error Commented Jan 17, 2016 at 16:58
  • @baldr also why in console even debug_task from celery.py is not showed? Commented Jan 17, 2016 at 17:28

2 Answers 2

Reset to default
3

Call your celery worker with explicit path to settings in the command line:

celery worker -A firstapp.celery

It seems the root cause is:

  • Celery connects to default queue and picks up a message
  • This message contains task name vk_wall.tasks.add
  • Celery tries to find this task in the list of known tasks and fails

This happens because you did not specify the settings source for celery. It should load Django project's settings or separate settings file.

Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

When do celery worker -A firstapp.settings -l info it throws long error ending ' ImportError: No module named 'firstapp' '. Added above setting.py data related to celery config
Try to find out where in your source code you try to import firstapp. Or show more exception lines
updated once more: added celery.py file which call firstapp.settings
from /var/www/firstapp which is root of django project
wait.. From your output I see that you're running it from /var/www/engine. Does it contain firstapp folder?
|
2

Fixed! The right call is:

celery worker -A engine -l debug

Also, this call is working if in settings.py i have changed line to be

CELERY_IMPORTS=["vk_wall.tasks"]

and in celery.py (engine instead of firstapp):

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'engine.settings')

from django.conf import settings  # noqa

app = Celery('engine')
Improve this answer

1 Comment

I can guess that celery.py is no more used in your project. It will be used only if you import it explicitely.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.