Celery tasks functions - web server vs remote server

Question

I'm willing to send tasks from a web server (running Django) to a remote machine that is holding a Rabbitmq server and some workers that I implemented with Celery.

If I follow the Celery way to go, it seems I have to share the code between both machines, which means replicating the workers logic code in the web app code.

So:

Is there a best practice to do that? Since code is redundant, I am thinking about using a git submodule (=> replicated in the web app code repo, and in the workers code repo)
Should I better use something else than Celery then?
Am I missing something?

ant31 · Accepted Answer · 2015-02-21 01:57:49Z

3

You can use send_task. It takes same parameters than apply_async but you only have to give the task name. Without loading the module in django you can send tasks:

app.send_task('tasks.add', args=[2, 2], kwargs={})

http://celery.readthedocs.org/en/latest/reference/celery.html#celery.Celery.send_task

answered Feb 21, 2015 at 1:57

ant31

4,8204 gold badges18 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

lajarre Over a year ago

Indeed that's what I was looking for. Yet it would be nice to describe a good practice about how to share the "task names" (dotted paths to tasks functions)

ant31 Over a year ago

yes by default the task name is the full python module path: "myproject.jobs.my_task". You can rename it using the decorator but I personnaly prefere default.

dm03514 · Accepted Answer · 2015-02-18 19:37:05Z

2

One way to manage this is to store your workers in your django project. Django and celery play nice to each other allowing you to use parts of your django project in your celery app. http://celery.readthedocs.org/en/latest/django/first-steps-with-django.html

Deploying this would mean that your web application would not use the modules involved with your celery workers, and on your celery machine your django views and such would never be used. This usually only results in a couple of megs of unused django application code...

answered Feb 18, 2015 at 19:37

dm03514

56.2k18 gold badges117 silver badges147 bronze badges

Collectives™ on Stack Overflow

Celery tasks functions - web server vs remote server

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related