using class methods as celery tasks

Question

I'm trying to use the methods of class as the django-celery tasks, marking it up using @task decorator. The same situation is discribed here, asked by Anand Jeyahar. It's something like this

class A:
    @task
    def foo(self, bar):
        ...

def main():
    a = A()
    ...
    # what i need
    a.foo.delay(bar) # executes as celery task 
    a.foo(bar) # executes locally

The problem is even if i use class instance like this a.foo.delay(bar) it says, that foo needs at least two arguments, which meens that self pointer misses.

More information:

I can't convert class to module because of inheritance
Methods are strongly depended on class members, so i can't make them static
Marking class as the task with @task decorator makes the class a task itself, and it could be possible to execute the methods from run() method, using some argument as a key for method selection, but it's not exactly what i want.
Creating an instance of class and passing it as self argument to methods changes the way i execute the methods not as celery taks, but as usual methods (i.e. while testing)
I've tried to find out how i can register the task dinamically, from constructor for example, but celery shares the code between the workers, so that's why it seems to be impossible.

Thanks for your help!

a = A() a.method(1,2) or a.method.delay(1,2) -- the result is the same — Thorin Schiffer
– Thorin Schiffer, Commented Feb 14, 2012 at 14:53

Espoir Murhabazi · Accepted Answer · 2020-05-22 20:45:48Z

55

Celery has experimental support for using methods as tasks since version 3.0.

The documentation for this is in celery.contrib.methods, and also mentions some caveats you should be aware of:

https://docs.celeryproject.org/en/3.1/reference/celery.contrib.methods.html

Be aware: support for contrib.methods removed from Celery since 4.0

edited May 22, 2020 at 20:45

Espoir Murhabazi

6,4415 gold badges49 silver badges78 bronze badges

answered Feb 13, 2012 at 13:36

asksol

19.6k5 gold badges67 silver badges69 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

ThiefMaster Over a year ago

Would be great to see this in a future version!

asksol Over a year ago

@ThiefMaster This is already in Celery 3.0 actually, see celery.contrib.methods: docs.celeryproject.org/en/latest/reference/…

ThiefMaster Over a year ago

Ah nice, consider updating your answer to include this.

Hamy Over a year ago

Please note: This appears to be removed in Oct 2014 due to it being too buggy to be useful - github.com/celery/celery/commit/…

snakecharmerb · Accepted Answer · 2021-01-26 09:22:38Z

11

Jeremy Satterfield has a clean and straight forward tutorial to write class based tasks if that's what you want to accomplish. You can check it here.

The magic is basically extending celery.Task class including a run() method, like something like this:

from celery import Task

class CustomTask(Task):
    ignore_result = True

    def __init__(self, arg):
        self.arg = arg

    def run(self):
        do_something_with_arg(self.arg)

and then run the task like this:

your_arg = 3

custom_task = CustomTask()
custom_task.delay(your_arg)

I am not sure if ignore_result = True part is necessary or not.

edited Jan 26, 2021 at 9:22

snakecharmerb

57.1k13 gold badges136 silver badges200 bronze badges

answered Apr 2, 2015 at 22:26

Gokhan Sari

8,1245 gold badges37 silver badges38 bronze badges

3 Comments

Sheesh Mohsin Over a year ago

I tried this, but its not working, its throwing this error - AttributeError: FileTask instance has no attribute 'delay'

Patlola Praveen Over a year ago

you should not override init in the classbased task. @SheeshMohsin please remove init and directly get arg in run function def run(self, arg): do_something_with_arg(arg)

kta Over a year ago

I think you still need to register the task to celery explicitly

mihaicc · Accepted Answer · 2012-07-03 18:22:25Z

5

When you have:

    a = A()

you can do:

    A.foo.delay(a, param0, .., paramN)

Cheers

answered Jul 3, 2012 at 18:22

mihaicc

3,1221 gold badge26 silver badges20 bronze badges

1 Comment

Asif Ali Over a year ago

How to do the same with apply_async, just to get the syntax at one place?

Wilmer Gonzalez · Accepted Answer · 2020-11-11 08:06:18Z

2

I ran into a similar situation and decided to wrap class methods within a simple function that would redirect its parameters to an instance of the class and it's execution of such methods:

class A:
    def foo(self, bar):
       # do this

a = A()

@app.task
def a_wrapper(bar):
    return a.foo(bar)

# probably in a different size with an import in-place:

a_wrapper.delay(bar)

answered Nov 11, 2020 at 8:06

Wilmer Gonzalez

291 bronze badge

Comments

Johnny Wellington · Accepted Answer · 2016-10-11 21:18:16Z

1

For me the only one that works is celery.current_app because just this passes self to the method.

So this should look like this:

from celery import current_app
from celery.contrib.methods import task_method

class A:
@current_app.task(filter=task_method, name='A.foo')
def foo(self, bar):
    ...

The name must be used if you have method with the same name in different classes.

edited Oct 11, 2016 at 21:18

answered Aug 27, 2014 at 13:44

Johnny Wellington

1392 silver badges5 bronze badges

1 Comment

Philipp Zettl Over a year ago

as @asksol mentioned this is deprecated since celery version 4 docs.celeryproject.org/en/latest/history/…

Collectives™ on Stack Overflow

using class methods as celery tasks

5 Answers 5

4 Comments

3 Comments

1 Comment

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

4 Comments

3 Comments

1 Comment

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related