0

I am new to celery, and right now I am using it compute Django tasks in the background. The problem I am having can be demonstrated using the below class method, where I am defining 3 functions inside TestCelery.

from celery import shared_task, chain, group, chord, signature
#from oralvis.celery import app
from django.conf import settings
from modules.SVD.createVideo import CreateVideo
import os
import time
import numpy as np
from oralvis.celery import app

class TestCelery:
    @app.task(bind=True)
    def do_work1(self, value):
        self.value = value
        value = value * 2
        value = np.array(value)
        print('do_work1', value)
        return value

    @app.task(bind=True)
    def do_work2(self, list_of_numpyArrays):
        self.do_work3()  # Call the instance method using self
        print('process do work 2')
        print(list_of_numpyArrays)
        return {'status': True}

    def do_work3(self):
        print('from do work 3')
        print(self.value)

# Create a shared instance of TestCelery
testCeleryInstance = TestCelery()

@shared_task
def testCelery(value):
    # Create a group of do_work1 tasks using signatures
    ret = chain(
        (group(testCeleryInstance.do_work1.s(i) for i in range(value))),
        testCeleryInstance.do_work2.s()).apply_async()

when I run the above code I get this error, which I don't understand. Is it that when celery runs a function inside a class, it does not have access to other methods inside the same class?

    [2023-09-29 20:52:17,008: WARNING/MainProcess] /home/ubuntu/webapp/oralvis/env/lib/python3.10/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
  warnings.warn(

[2023-09-29 20:52:17,010: INFO/MainProcess] mingle: searching for neighbors
[2023-09-29 20:52:18,017: INFO/MainProcess] mingle: all alone
[2023-09-29 20:52:18,036: INFO/MainProcess] celery@ip-10-79-34-9 ready.
[2023-09-29 20:52:30,464: INFO/MainProcess] Task SVDExperimentCompare.tasks.testCelery[ab950d29-bcdb-40db-96d3-8920894e734d] received
[2023-09-29 20:52:30,485: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[fbb0a44d-23ef-4b92-a864-76e205ac7f5c] received
[2023-09-29 20:52:30,491: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[395845a7-e19c-467b-8e10-f3815333d961] received
[2023-09-29 20:52:30,495: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[403c9a79-c3e0-4528-8e37-7d1ac5ab2f80] received
[2023-09-29 20:52:30,499: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[abfb7a67-5601-4ba6-8fb5-f7d2ccdbd9c3] received
[2023-09-29 20:52:30,505: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[a647471f-a38c-42aa-b95a-4b62ac5cb281] received
[2023-09-29 20:52:30,508: INFO/MainProcess] Task SVDExperimentCompare.tasks.testCelery[ab950d29-bcdb-40db-96d3-8920894e734d] succeeded in 0.038984907005215064s: None
[2023-09-29 20:52:30,512: WARNING/MainProcess] do_work1
[2023-09-29 20:52:30,512: WARNING/MainProcess]  
[2023-09-29 20:52:30,515: WARNING/MainProcess] 4
[2023-09-29 20:52:30,521: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[403c9a79-c3e0-4528-8e37-7d1ac5ab2f80] succeeded in 0.0104450340004405s: array(4)
[2023-09-29 20:52:30,528: WARNING/MainProcess] do_work1
[2023-09-29 20:52:30,528: WARNING/MainProcess]  
[2023-09-29 20:52:30,528: WARNING/MainProcess] 6
[2023-09-29 20:52:30,533: WARNING/MainProcess] do_work1
[2023-09-29 20:52:30,534: WARNING/MainProcess]  
[2023-09-29 20:52:30,534: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[abfb7a67-5601-4ba6-8fb5-f7d2ccdbd9c3] succeeded in 0.01316564499575179s: array(6)
[2023-09-29 20:52:30,535: WARNING/MainProcess] 0
[2023-09-29 20:52:30,538: WARNING/MainProcess] do_work1
[2023-09-29 20:52:30,538: WARNING/MainProcess]  
[2023-09-29 20:52:30,538: WARNING/MainProcess] 8
[2023-09-29 20:52:30,540: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[fbb0a44d-23ef-4b92-a864-76e205ac7f5c] succeeded in 0.05309743800171418s: array(0)
[2023-09-29 20:52:30,541: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[a647471f-a38c-42aa-b95a-4b62ac5cb281] succeeded in 0.0055748630038578995s: array(8)
[2023-09-29 20:52:30,543: WARNING/MainProcess] do_work1
[2023-09-29 20:52:30,543: WARNING/MainProcess]  
[2023-09-29 20:52:30,543: WARNING/MainProcess] 2
[2023-09-29 20:52:30,548: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work2[de14846d-7c2d-43b0-9e83-50988d0cf8c0] received
[2023-09-29 20:52:30,549: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[395845a7-e19c-467b-8e10-f3815333d961] succeeded in 0.0558565669998643s: array(2)
[2023-09-29 20:52:30,553: ERROR/MainProcess] Task SVDExperimentCompare.tasks.do_work2[de14846d-7c2d-43b0-9e83-50988d0cf8c0] raised unexpected: AttributeError("'do_work2' object has no attribute 'do_work3'")
Traceback (most recent call last):
  File "/home/ubuntu/webapp/oralvis/env/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/ubuntu/webapp/oralvis/env/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/ubuntu/webapp/oralvis/oralvis/SVDExperimentCompare/tasks.py", line 21, in do_work2
    self.do_work3()  # Call the instance method using self
AttributeError: 'do_work2' object has no attribute 'do_work3'
2
  • shouldn't it be TestCelery instead of testCelery on the last rows? or maybe even testCeleryInstance? Don't know celery, but logic tells me that problem in object to which you refer Commented Sep 29, 2023 at 20:48
  • sorry I just edited the code, but it is still giving me error... Commented Sep 29, 2023 at 20:55

1 Answer 1

0

The error is pretty obvious: it looks for attribute do_work3 in object do_work2. Why it happens:

from the docs (https://docs.celeryq.dev/en/stable/userguide/tasks.html#bound-tasks):

A task being bound means the first argument to the task will always be the task instance (self)

i.e. it will be a task instance, not an instance of your class TestCelery.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.