Celery task best practices in Django/Python

Question

Consider a function that requires some heavy lifting to be done asynchronously. Calling clients can either recive a cached version immediately, or recive a response that the numbers are being crunched (a valid response for the clients).

Is the following snippet a sound implementation for this pattern?

from django.core import cache
from proj.celery import app

class SomeModel(models.Model):
    # ...
    def get_crunched_numbers(self):
        cache_key = 'foo:{}'.format(self.id)
        res = cache.get(cache_key)
        if not res:
            @app.task
            def gen_crunched_numbers():
                res = do_heavy_lifting()
                cache.set(cache_key, res)
                return res
            gen_crunched_numbers.delay()
            return 'crunching... come back later'
        else:
            return res

Are there better alternatives to running Celery tasks like so, while containing all the logic in a single piece of code?

Edit: as mentioned in the comments, this code doesn't even work. So any suggestions on a nice pattern are much obliged.

Does that even work at all? I would be surprised if a nested function like that could be serialized. Much better surely to make cache_key a parameter to a completely standalone function. — Daniel Roseman
– Daniel Roseman, Commented Aug 3, 2014 at 14:09
@DanielRoseman you are correct this only worked locally (when using ALWAYS_EAGER=True) and is totally unserializable. — Yuval Adam
– Yuval Adam, Commented Aug 3, 2014 at 15:45

daniula · Accepted Answer · 2014-08-03 20:30:52Z

2

Your code looks very confusing. Why don't you define celery task function outside of your class and call it like this:

from django.core import cache
from proj.celery import app

class SomeModel(models.Model):
    # ...
    def get_crunched_numbers(self):
        cache_key = 'foo:{}'.format(self.id)
        res = cache.get(cache_key)
        if not res:
            gen_crunched_numbers.delay(cache_key)
            return 'crunching... come back later'
        else:
            return res

@app.task
def gen_crunched_numbers(cache_key):
    res = do_heavy_lifting()
    cache.set(cache_key, res)
    return res

Also I usually prefer to create tasks with bind=True:

@app.task(bind=True)
def gen_crunched_numbers(self, cache_key):
    res = do_heavy_lifting()
    cache.set(cache_key, res)
    return res

Which give me access to task context through self.request. For example to change behavior depending if function is called through celery or directly:

@app.task(bind=True)
def gen_crunched_numbers(self, cache_key):
    res = do_heavy_lifting()
    cache.set(cache_key, res)
    if self.request.called_directly:
        return res
    else:
        return { 'result': res, 'cache': cache_key }

answered Aug 3, 2014 at 20:30

daniula

7,0384 gold badges35 silver badges49 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Yuval Adam Over a year ago

I was trying to encapsulate all the logic in a single function, but as you saw, it doesn't really work. Your suggestion looks good, and it's what I'm currently using. The bind=True is a very good improvement and I'll likely use it as well. Thanks!

Collectives™ on Stack Overflow

Celery task best practices in Django/Python

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related