Add functions dynamically to existing celery worker processes?

Question

I'm getting started with celery and I want to know if it is possible to add modules to celeryd processes that have already been started. In other words, instead of adding modules via celeryconfig.py as in

CELERY_IMPORTS = ("tasks", "additional_module" )

before starting the workers, I want to make additional_module available later somehow after the worker processes have started.

thanks in advance.

rlotun · Accepted Answer · 2011-01-18 16:51:02Z

2

You can achieve your goal by starting a new celeryd with an expanded import list and eventually gracefully shutting down your old worker (after it's finished its current jobs).

Because of the asynchronous nature of getting jobs pushed to you and only marking them done after celery has finished its work, you won't actually miss any work doing it this way. You should be able to run the celery workers on the same machine - they'll simply show up as new connections to RabbitMQ (or whatever queue backend you use).

edited Jan 18, 2011 at 16:51

answered Jan 17, 2011 at 11:29

rlotun

8,1034 gold badges30 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

reckoner Over a year ago

how would I start a new celery worker from within the client? I think you can shutdown workers from inside the client, but how do you start one?

rlotun Over a year ago

I didn't clearly state what I meant - I should have said start a new celery daemon (i.e. celeryd). I've updated my post.

Thomas Browne Over a year ago

Is this advice still relevant / best practise, as I write this, Aug 2014, or have better ways come along? I have many custom functions that I am trying for numerical optimization which takes ages (therefore using Celery to run them on numerous computers), so I would like to send new functions to all the workers, periodically, increasing the range of different optimization functions available to each worker, on the fly. I've read about module reload on the fly but this seems a bit clunky? Can I just send new functions over periodically, in say, a dict, guaranteeing all workers will get them?

Collectives™ on Stack Overflow

Add functions dynamically to existing celery worker processes?

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related