Module importing multiple times with django + celery

Question

I have a module which is expensive to import (it involves downloading a ~20MB index file), which is used by a celery worker. Unfortunately I can't figure out how to have the module imported only once, and only by the celery worker.

Version 1 tasks.py file:

import expensive_module

@shared_task
def f():
    expensive_module.do_stuff()

When I organize the file this way the expensive module is imported both by the web server and the celery instance, which is what I'd expect since the tasks module is imported in both and they're difference processes.

Version 2 tasks.py file:

@shared_task:
def f():
    import expensive_module
    expensive_module.do_stuff()

In this version the web server never imports the module (which is good), but the module gets re-imported by the celery worker every time f.delay() is called. This is what really confuses me. In this scenario, why is the module re-imported every time this function is run by the celery worker? How can I re-organize this code to have only the celery worker import the expensive module, and have the module imported only once?

As a follow-on, less important question, in Version 1 of the tasks.py file, why does the web instance import the expensive module twice? Both times it's imported form urls.py when django runs self._urlconf_module = import_module(self.urlconf_name).

Thing is celery tasks are ideally idempotent meaning they don't have any state and always deliver the same output when called with the same parameters. - You are somewhat introducing state, thus doing this with celery might become difficult. — Christine Schäpers
– Christine Schäpers, Commented Aug 10, 2015 at 19:22

Zhack · Accepted Answer · 2015-10-23 22:20:55Z

0

Make a duplicate tasks.py file for webserver which has empty tasks and no unneeded imports.

For celery use version 1 where you import only once instead of every time you call that task.

Been there and it works.

answered Oct 23, 2015 at 22:20

Zhack

967 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Module importing multiple times with django + celery

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related