I am new to celery, and right now I am using it compute Django tasks in the background. The problem I am having can be demonstrated using the below class method, where I am defining 3 functions inside TestCelery.
from celery import shared_task, chain, group, chord, signature
#from oralvis.celery import app
from django.conf import settings
from modules.SVD.createVideo import CreateVideo
import os
import time
import numpy as np
from oralvis.celery import app
class TestCelery:
@app.task(bind=True)
def do_work1(self, value):
self.value = value
value = value * 2
value = np.array(value)
print('do_work1', value)
return value
@app.task(bind=True)
def do_work2(self, list_of_numpyArrays):
self.do_work3() # Call the instance method using self
print('process do work 2')
print(list_of_numpyArrays)
return {'status': True}
def do_work3(self):
print('from do work 3')
print(self.value)
# Create a shared instance of TestCelery
testCeleryInstance = TestCelery()
@shared_task
def testCelery(value):
# Create a group of do_work1 tasks using signatures
ret = chain(
(group(testCeleryInstance.do_work1.s(i) for i in range(value))),
testCeleryInstance.do_work2.s()).apply_async()
when I run the above code I get this error, which I don't understand. Is it that when celery runs a function inside a class, it does not have access to other methods inside the same class?
[2023-09-29 20:52:17,008: WARNING/MainProcess] /home/ubuntu/webapp/oralvis/env/lib/python3.10/site-packages/celery/worker/consumer/consumer.py:507: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
whether broker connection retries are made during startup in Celery 6.0 and above.
If you wish to retain the existing behavior for retrying connections on startup,
you should set broker_connection_retry_on_startup to True.
warnings.warn(
[2023-09-29 20:52:17,010: INFO/MainProcess] mingle: searching for neighbors
[2023-09-29 20:52:18,017: INFO/MainProcess] mingle: all alone
[2023-09-29 20:52:18,036: INFO/MainProcess] celery@ip-10-79-34-9 ready.
[2023-09-29 20:52:30,464: INFO/MainProcess] Task SVDExperimentCompare.tasks.testCelery[ab950d29-bcdb-40db-96d3-8920894e734d] received
[2023-09-29 20:52:30,485: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[fbb0a44d-23ef-4b92-a864-76e205ac7f5c] received
[2023-09-29 20:52:30,491: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[395845a7-e19c-467b-8e10-f3815333d961] received
[2023-09-29 20:52:30,495: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[403c9a79-c3e0-4528-8e37-7d1ac5ab2f80] received
[2023-09-29 20:52:30,499: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[abfb7a67-5601-4ba6-8fb5-f7d2ccdbd9c3] received
[2023-09-29 20:52:30,505: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[a647471f-a38c-42aa-b95a-4b62ac5cb281] received
[2023-09-29 20:52:30,508: INFO/MainProcess] Task SVDExperimentCompare.tasks.testCelery[ab950d29-bcdb-40db-96d3-8920894e734d] succeeded in 0.038984907005215064s: None
[2023-09-29 20:52:30,512: WARNING/MainProcess] do_work1
[2023-09-29 20:52:30,512: WARNING/MainProcess]
[2023-09-29 20:52:30,515: WARNING/MainProcess] 4
[2023-09-29 20:52:30,521: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[403c9a79-c3e0-4528-8e37-7d1ac5ab2f80] succeeded in 0.0104450340004405s: array(4)
[2023-09-29 20:52:30,528: WARNING/MainProcess] do_work1
[2023-09-29 20:52:30,528: WARNING/MainProcess]
[2023-09-29 20:52:30,528: WARNING/MainProcess] 6
[2023-09-29 20:52:30,533: WARNING/MainProcess] do_work1
[2023-09-29 20:52:30,534: WARNING/MainProcess]
[2023-09-29 20:52:30,534: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[abfb7a67-5601-4ba6-8fb5-f7d2ccdbd9c3] succeeded in 0.01316564499575179s: array(6)
[2023-09-29 20:52:30,535: WARNING/MainProcess] 0
[2023-09-29 20:52:30,538: WARNING/MainProcess] do_work1
[2023-09-29 20:52:30,538: WARNING/MainProcess]
[2023-09-29 20:52:30,538: WARNING/MainProcess] 8
[2023-09-29 20:52:30,540: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[fbb0a44d-23ef-4b92-a864-76e205ac7f5c] succeeded in 0.05309743800171418s: array(0)
[2023-09-29 20:52:30,541: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[a647471f-a38c-42aa-b95a-4b62ac5cb281] succeeded in 0.0055748630038578995s: array(8)
[2023-09-29 20:52:30,543: WARNING/MainProcess] do_work1
[2023-09-29 20:52:30,543: WARNING/MainProcess]
[2023-09-29 20:52:30,543: WARNING/MainProcess] 2
[2023-09-29 20:52:30,548: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work2[de14846d-7c2d-43b0-9e83-50988d0cf8c0] received
[2023-09-29 20:52:30,549: INFO/MainProcess] Task SVDExperimentCompare.tasks.do_work1[395845a7-e19c-467b-8e10-f3815333d961] succeeded in 0.0558565669998643s: array(2)
[2023-09-29 20:52:30,553: ERROR/MainProcess] Task SVDExperimentCompare.tasks.do_work2[de14846d-7c2d-43b0-9e83-50988d0cf8c0] raised unexpected: AttributeError("'do_work2' object has no attribute 'do_work3'")
Traceback (most recent call last):
File "/home/ubuntu/webapp/oralvis/env/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/ubuntu/webapp/oralvis/env/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__
return self.run(*args, **kwargs)
File "/home/ubuntu/webapp/oralvis/oralvis/SVDExperimentCompare/tasks.py", line 21, in do_work2
self.do_work3() # Call the instance method using self
AttributeError: 'do_work2' object has no attribute 'do_work3'