2

I have been scouring the Internet for some sort of guide for setting up django, celery, and redis so the celery task runs on a remote server, and I cannot seem to find out how to do this.

I have three machines - tsunami (development), trout (production), and t-rex (Nvidia card). I am running some image manipulation algorithms using celery and redis. I have the same django/celery code base on all three computers. When I run the algorithms on tsunami or trout without CUDA support, it takes hours to complete. When I run the same algorithms on t-rex with CUDA support, they complete in minutes. What I would like to do is offload the celery tasks from trout and tsunami to t-rex, and return the results to the either trout or tsunami. By results, I mean a file to save on the local file system or data added to the local mysql (8.0.32) database. Is this even possible?

Can someone point me to a guide on how to set this up? I am running django 4.1.7, celery 5.2.7 and redis server 6.0.16. Trout and tsunami are running Ubuntu 20.04, and t-rex is Ubuntu 22.04. All are running Python 3.8.

I am concerned that when I install dlib on trout and tsunami, it is compiled without CUDA support because there are no Nvidia cards. All machines are running dlib 19.24.0, but only t-rex has it compiled with CUDA support. Is this a show stopper for my plan to use t-rex as a remote CUDA processor for the other machines?

Thanks!

Improve this question

2 Answers 2

Reset to default
2

You need setup different task queues and routes for your servers.

Check out the celery docs on how to setup task routing

To specify routes define task_routes. Let's say you have your task that runs image manipulation defined under the module feed.tasks, use something like the following in your celery configuration to ensure that those tasks are only sent to the feeds queue:

app.conf.task_routes = {'feed.tasks.*': {'queue': 'feeds'}}

To specify what queues to consume from, modify your celery worker script and give a comma separated list of queues using the -Q option:

$ celery -A proj worker -l INFO -Q foo,bar,baz,feeds

By default celery worker will read from all queues. You can exclude queues on tsunami and trout, using the -X option

$ celery -A proj worker -l INFO -X feeds
Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Thanks! After reading this post, stackoverflow.com/questions/23551808/…, it seems I have to have (1) a shared file system for the celery workers to read/write data and (2) a shared database for the celery workers to read and write. Is that true?
After looking at the task routing page, I don't see how to route tasks to different redis servers. If tsunami has to process an image, don't I have to send that task to the redis server on t-rex so a celery task there can do the processing?
You will need some way to transmit the results back to trout or tsunami, ex shared file system, shared database, etc. would work. In order to get your task queues shared across servers, you need to point your CELERY_BROKER_URL on all of your servers to the same redis server.
1

Here servers can be docker images (docker-compose) or containers or pods (on k8s).
Common tasks on all servers

  1. copy the project
  2. pip install -r requirements.txt

server A: run django server as python manage.py runserver (if using uvicorn or gunicorn, then run the appropriate app server accordingly).

server B: celery -A proj_name worker

server C: celery -A proj_name worker -l INFO -Q foo,bar # logging level is Info and run tasks on queue foo and bar.

server D: celery -A proj_name worker -l INFO -Q email,url # logging level is Info and runs tasks on queue email and url.

Btw the location to start the django server and celery worker is the same, that is the project's root directory, that is the directory where the manage.py is located. If you have multiple remote executors, for the sake of redundancy and high availability or fault tolerance, it is recommended to have a queue run on more than 1 worker. ex:-

server B: celery -A proj_name worker -l INFO -Q foo,bar
server C: celery -A proj_name worker -l INFO -Q bar,email
server D: celery -A proj_name worker -l INFO -Q email,foo
server B,C both process queue bar.
server C,D both process queue email.
server D,B both process queue bar.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.