Celery 4.2-Django RecursionError: maximum recursion depth exceeded

Question

My __init__.py file

from __future__ import absolute_import, unicode_literals
from .celery import app as celery_app

__all__ = ('celery_app',)

My celery.py file

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')
app = Celery('myproject')

app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

my /etc/supervisor/conf.d/celery.conf file:

[program:]
command=/home/user/venv/bin/celery -A myproject worker --loglevel=INFO
environment=PYTHONPATH=/home/user/project-folder
user=user
numprocs=1
autostart=true
autorestart=true
stdout_logfile=/home/user/logs/celery.log
stderr_logfile=/home/user/logs/celery.log
startssecs=10
stopwaitsecs = 600
stopasgroup=true
priority=100

and this is the last section of the error message in celery.log

File "/home/user/venv/lib/python3.6/site-packages/celery/loaders/base.py", line 131, in config_from_object
   self._conf = force_mapping(obj)
File "/home/user/venv/lib/python3.6/site-packages/celery/utils/collections.py", line 48, in force_mapping
   return DictAttribute(m) if not isinstance(m, Mapping) else m
File "/home/user/venv/lib/python3.6/abc.py", line 183, in __instancecheck__
   subclass = instance.__class__
File "/home/user/venv/lib/python3.6/site- packages/django/utils/functional.py", line 215, in inner
   self._setup()
File "/home/user/venv/lib/python3.6/site-packages/django/conf/__init__.py", line 43, in _setup
   self._wrapped = Settings(settings_module)
File "/home/user/venv/lib/python3.6/site-packages/django/conf/__init__.py", line 106, in __init__
   mod = importlib.import_module(self.SETTINGS_MODULE)
File "/home/user/venv/lib/python3.6/importlib/__init__.py", line 126, in import_module
   return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 994, in _gcd_import
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 951, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 894, in _find_spec
File "<frozen importlib._bootstrap_external>", line 1157, in find_spec
File "<frozen importlib._bootstrap_external>", line 1129, in _get_spec
File "<frozen importlib._bootstrap_external>", line 1273, in find_spec
File "<frozen importlib._bootstrap_external>", line 1231, in _get_spec
File "<frozen importlib._bootstrap_external>", line 556, in 
spec_from_file_location
RecursionError: maximum recursion depth exceeded

I don't know whats wrong with this configuration?

I am using supervisor to start the gunicorn and celery in two separate conf files. Gunicorn runs well but my celery workers don't. The command to start the celery workers is the first line in my celery.conf file — Ronny Alfonso
– Ronny Alfonso, Commented Aug 11, 2018 at 13:33
Can you make sure your priority is different on gunicorn? I think gunicorn should be first, then celery — wdfc
– wdfc, Commented Aug 12, 2018 at 3:00
@JerinPeterGeorge and @wdfc that's actually what is happening. I also have my gunicorn.conf file. When I run sudo supervisorctl reread and then sudo supervisorctl update the gunicorn daemon is run first, and then it fails with celery.conf, with the above message — Ronny Alfonso
– Ronny Alfonso, Commented Aug 12, 2018 at 13:33
too late for you, but maybe helpful for someone else. Make sure the environment variables set in your gunicorn.service are also set in the celery.service and clean up any old files in pycache after hours of trying things. — AMG
– AMG, Commented May 31, 2020 at 3:18

Kishor Pawar · Accepted Answer · 2020-04-08 15:34:12Z

7

I also have the same problem, when I delete the following sentence

 app.config_from_object('django.conf:settings', namespace='CELERY')

then it works. So I think the problem is happend because of that. Maybe we should specify the config of this function.

edited Apr 8, 2020 at 15:34

Kishor Pawar

3,5463 gold badges34 silver badges64 bronze badges

answered Aug 5, 2019 at 9:30

Zoupers

711 silver badge2 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Zoupers Over a year ago

oh , I'm sorry to say that the sentence is app.config_from_object('django.conf:settings', namespace='CELERY'), not app.auto...

Kishor Pawar Over a year ago

Somehow that works, but don't know why? Perhaps, the official document says to have that line.

Kishor Pawar · Accepted Answer · 2020-04-11 09:30:35Z

4

from django.conf import settings

app.config_from_object(settings, namespace='CELERY')

Actually this settings go in there.

answered Apr 11, 2020 at 9:30

Kishor Pawar

3,5463 gold badges34 silver badges64 bronze badges

Comments

2ps · Accepted Answer · 2018-08-14 04:03:49Z

1

You shouldn't be importing your celery app in your __init__.py file. Take that out and it will fix your infinite startup import loop. Essentially, your celery app kicks off the django settings module initialization which in turn loads the apps and tries to reload the project __init__.py file which loads your celery file, which then tries to reload the settings file because it hasn't initialized which in turn . . . (you get the idea).

answered Aug 14, 2018 at 4:03

2ps

16k2 gold badges34 silver badges53 bronze badges

1 Comment

Ronny Alfonso Over a year ago

I keep having the same error. I had no choice, I've spent 2 weeks banging my head to the desk and I had to do a workaround: def threader(task,*args): t = threading.Thread(target=task, args=args) t.daemon = True t.start(). With this patch I call the method and its parameters and that does my email stuff. I put this function inside a utils.py file I have in my project

Andrey Sh · Accepted Answer · 2019-08-19 17:52:11Z

1

In case it helps someone - I had a similar problem and it was caused by an exception in the overridden on_configure method of Celery class. I guess anything that prevents Celery._load_config method from executing normally might cause the "maximum recursion depth exceeded" error.

answered Aug 19, 2019 at 17:52

Andrey Sh

1267 bronze badges

Comments

Benny Chan · Accepted Answer · 2022-07-06 11:57:39Z

0

I try to comment the line

app.config_from_object('django.conf:settings', namespace='CELERY')

and the error is gone. When the celery-related settings are not found in settings.py, it gives me this error. After that, I add

CELERY_BROKER_URL = 'redis://redis:6379'

CELERY_RESULT_BACKEND = 'redis://redis:6379'

It works again. I'm using docker compose to set up django + redis + celery. I guess it cannot detect the settings.py file without any celery settings

answered Jul 6, 2022 at 11:57

Benny Chan

9321 gold badge10 silver badges28 bronze badges

Collectives™ on Stack Overflow

Celery 4.2-Django RecursionError: maximum recursion depth exceeded

5 Answers 5

2 Comments

Comments

1 Comment

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

2 Comments

Comments

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related