Django celery cannot import name 'Celery' from 'celery' after rebuilding Dockerfile

Question

I run Django and Celery in Docker. It's working fine, but after I restart Docker, celery can't start because of import name error.

Errors below:

v_web            | Traceback (most recent call last):
v_web            |   File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 354, in run_from_argv
v_web            |     self.execute(*args, **cmd_options)
v_web            |   File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 393, in execute
v_web            |     self.check()
v_web            |   File "/usr/local/lib/python3.7/site-packages/django/core/management/base.py", line 423, in check
v_web            |     databases=databases,
v_web            |   File "/usr/local/lib/python3.7/site-packages/django/core/checks/registry.py", line 76, in run_checks
v_web            |     new_errors = check(app_configs=app_configs, databases=databases)
v_web            |   File "/usr/local/lib/python3.7/site-packages/django/core/checks/urls.py", line 100, in check_url_settings
v_web            |     value = getattr(settings, name)
v_web            |   File "/usr/local/lib/python3.7/site-packages/django/conf/__init__.py", line 82, in __getattr__
v_web            |     self._setup(name)
v_web            |   File "/usr/local/lib/python3.7/site-packages/django/conf/__init__.py", line 69, in _setup
v_web            |     self._wrapped = Settings(settings_module)
v_web            |   File "/usr/local/lib/python3.7/site-packages/django/conf/__init__.py", line 170, in __init__
v_web            |     mod = importlib.import_module(self.SETTINGS_MODULE)
v_web            |   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
v_web            |     return _bootstrap._gcd_import(name[level:], package, level)
v_web            |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
v_web            |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
v_web            |   File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
v_web            |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
v_web            |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
v_web            |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
v_web            |   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
v_web            |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
v_web            |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
v_web            |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
v_web            |   File "/app/notification/__init__.py", line 1, in <module>
v_web            |     from .celery import app as celery_app
v_web            |   File "/app/notification/celery.py", line 2, in <module>
v_web            |     from celery import Celery
v_web            | ImportError: cannot import name 'Celery' from 'celery' (/usr/local/lib/python3.7/site-packages/celery/__init__.py)

Here's my project structure:

notifications
   notification
      __init__.py
      celery.py

Here's my init.py file:

from .celery import app as celery_app
__all__ = ("celery_app",)

Here's my celery.py:

import os
from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "notification.settings")

app = Celery("notification")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

Here's my docker file:

version: "3.9"
   
services:

  redis:
    container_name: v_redis
    image: redis:alpine
    restart: always

  db:
    container_name: v_db
    image: postgres
    volumes:
      - ./data/db:/var/lib/postgresql/data
    env_file:
      - ./production.env
    restart: always

  web:
    container_name: v_web
    build: .
    command: bash -c "python manage.py makemigrations && python manage.py migrate && python manage.py collectstatic --no-input && gunicorn notification.wsgi:application --limit-request-line 0 --workers=4 --threads=4 --worker-class=gthread --bind 0.0.0.0:8000"
    env_file:
      - ./production.env
    volumes:
      - .:/app
      - ./staticfiles:/staticfiles
    expose:
      - 8000
    depends_on:
      - db
      - redis
    restart: always
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/"]
      interval: 30s
      timeout: 10s
      retries: 5

  celery:
    container_name: v_celery
    build: .
    command: celery -A notification worker -l info --concurrency 4
    env_file:
      - ./production.env
    # volumes:
    #   - ./redis/:/usr/src/app/
    depends_on:
      web:
        condition: service_healthy
      redis:
        condition: service_started
    restart: always


  celery-beat:
    container_name: v_celery_beat
    build: .
    command: celery -A notification beat -l INFO --scheduler django_celery_beat.schedulers:DatabaseScheduler
    env_file:
      - ./production.env
    # volumes:
    #   - ./redis/:/usr/src/app/
    depends_on:
      - redis
    restart: always

  flower:
    container_name: v_celery_flower
    image: mher/flower
    command: celery flower
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - FLOWER_PORT=5555
    ports:
      - 5555:5555
    depends_on:
      - redis
    restart: always

  nginx:
    container_name: v_nginx
    build: ./nginx
    volumes:
      - ./staticfiles:/staticfiles
      # - ./nginx/nginx.conf:/etc/nginx/nginx.conf
    ports:
      - 80:80
      - 443:443
    depends_on:
      - web
    restart: always

My Dockerfile:

# syntax=docker/dockerfile:1
FROM python:3.7
WORKDIR /app
COPY . /app/
RUN pip install -r requirements.txt

My requirements.txt:

Django>=3.0,<4.0
gunicorn==20.1.0
psycopg2-binary>=2.8
django-crispy-forms
django-celery-beat
django-select2
django-redis
slack_sdk
PyMuPDF
Pillow
google-api-python-client
google-auth-oauthlib
oauth2client
pandas
celery==5.1.2
redis==3.5.3
sqlalchemy
sshtunnel
matplotlib
boto3

I don't know why it fails after rebuilding Dockerfile. Please help. Thanks.

You should probably take a look at this stackoverflow.com/questions/19926750/… — WMRamadan
– WMRamadan, Commented Oct 3, 2022 at 10:44
@WMRamadan: I try to change the name of celery.py to other (like your link), but it doesn't work, still shows the same error. — Viet Pm
– Viet Pm, Commented Oct 3, 2022 at 11:09
try pip list if you have importlib-metadata installed ot not — monim
– monim, Commented Oct 3, 2022 at 11:29

monim · Accepted Answer · 2022-10-03 11:29:14Z

29

this error is a bug that occur only with python 3.7 (the same version you're using) because the latest release of importlib-metadata .

just downgrade importlib-metadata<5.0 : pip install importlib-metadata==4.13.0

answered Oct 3, 2022 at 11:29

monim

4,4762 gold badges14 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Viet Pm Over a year ago

I also guessed there's some conflict with libraries in python 3.7 and celery lib, because it still works in my local computer with python 3.8. So yesterday I switched from python 3.7 to python 3.8 in Dockerfile and it worked too. Now I know the real reason after reading your answer, thanks.

Collectives™ on Stack Overflow

Django celery cannot import name 'Celery' from 'celery' after rebuilding Dockerfile

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related