Problem running celery in a different docker container than the Flask app

Question

I’m running a Flask application with Celery for submitting sub-processes using docker-compose. However I cannot make Celery work when trying to run it in a different container.

If I run Celery in the same container I’m running the flask app it works, but feels like the wrong way, I’m coupling two different things in one container, by adding this in the startup script before the flask app runs:

nohup celery worker -A app.controller.engine.celery -l info &

However if I add Celery as a new container in my docker-compose.yml it doesn’t work. This is my config:

(..)

engine:
  image: engine:latest
  container_name: engine
  ports:
    - 5000:5000
  volumes:
    - $HOME/data/engine-import:/app/import
  depends_on:
    - mongo
    - redis
  environment:
    - HOST=localhost

celery:
  image: engine:latest
  environment:
        - C_FORCE_ROOT=true
  command: ["/bin/bash", "-c", "./start-celery.sh"]
  user: nobody
  depends_on:
    - redis

(..)

And this is the start-celery.sh:

#!/bin/bash
source ./env/bin/activate

cd ..
celery worker -A app.controller.engine.celery -l info

Its logs:

INFO:engineio:Server initialized for eventlet.
INFO:engineio:Server initialized for threading.
[2018-09-12 09:43:19,649: INFO/MainProcess] Connected to redis://redis:6379//
[2018-09-12 09:43:19,664: INFO/MainProcess] mingle: searching for neighbors
[2018-09-12 09:43:20,697: INFO/MainProcess] mingle: all alone
[2018-09-12 09:43:20,714: INFO/MainProcess] celery@8729618bd4bc ready.

And that’s all, processes are not submited to it.

What can be missing?

Does the code base live in the mapped volume? As I assume it's the same, should it not also be mapped into the celery container? — Bjoern Stiel
– Bjoern Stiel, Commented Sep 12, 2018 at 9:59
Should I map the code when I'm running celery using the same image? That line is not the code but an import folder for specific files that get imported through the app. — magnoz
– magnoz, Commented Sep 12, 2018 at 10:00
What I mean is, are you sure that the code base inside the celery container is exactly the same as the code base inside your engine container? Is there any more Celery log output (debug)? Can you get Celery to log the app's registered tasks on startup? — Bjoern Stiel
– Bjoern Stiel, Commented Sep 12, 2018 at 10:03
Can you bash into your celery container and try celery worker -A app.controller.engine.celery inspect registered to see if your tasks are registered? — Bjoern Stiel
– Bjoern Stiel, Commented Sep 12, 2018 at 10:19

magnoz · Accepted Answer · 2018-09-13 07:47:10Z

1

I've found that It works only if I add this to the docker-compose definition of the celery service:

environment: 
      - C_FORCE_ROOT=true

I wonder though why I didn't get any error otherwise.

edited Sep 13, 2018 at 7:47

answered Sep 12, 2018 at 11:34

magnoz

2,0055 gold badges26 silver badges46 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

nicorellius Over a year ago

Here we are in 2022, and it seems the best practice is to install Celery in the application container and run the worker with something like Supervisor. Have you changed you strategy for your app in the last 4 years? Just curious, as I've considered separating my Celery to its own image too, but wondered if it was the right call.

Collectives™ on Stack Overflow

Problem running celery in a different docker container than the Flask app

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related