1,746 questions
0
votes
1
answer
42
views
How to disable Celery startup logs?
I'm getting a bunch of logs like this:
[2025-11-29 16:13:15,731]
def group(self, tasks, result, group_id, partial_args, add_to_parent=0):
return 1
[2025-11-29 16:13:15,732]
def xmap(task, it):
...
- 630
1
vote
0
answers
51
views
Django Celery Beat SQS slow scheduling
Beat seems to be sending the messages into SQS very slowly, about 100/minute.
Every Sunday I have a sendout to about 16k users, and they're all booked for 6.30pm. Beat starts picking it up at the ...
- 11
0
votes
0
answers
175
views
KEDA + Celery: Need Immediate Pod Scaling for Each Queued Task (Zero Queue Length Goal)
I have KEDA + Celery setup working, but there's a timing issue with scaling. I need immediate pod scaling when tasks are queued - essentially maintaining zero pending tasks at all times by spinning up ...
- 31
0
votes
0
answers
68
views
Celery and AWS MQ Disconnections
I set up a celery + beat containers in ECS and used AWS MQ as the broker. However, my setup seems to be very prone to disconnections, where celery workers are suddenly disconnected from MQ.
I end up ...
- 79
0
votes
0
answers
10
views
Django Rest Upload Files files in the background
I have a function that uploads files to from a listings platform. Users can upload multiple files at once and each can be about 10mb in size
I have tried adding celery to upload the files in the ...
- 371
0
votes
0
answers
27
views
Celery tasks are not running
I'm working on a drf app where I need to use celery. I'm trying to run a simple task to make sure the set up is solid for later.
settings.py:
CELERY_BROKER_URL = 'redis://localhost:6379/0'
...
0
votes
0
answers
66
views
celery worker not consuming tasks from rabbitMQ queue
I have a queue in RabbitMQ broker called queue-ktm of type quorum. There are tasks pushed to this queue and available.
I run a celery worker to consume from this queue with following configuration:
...
- 4,161
0
votes
1
answer
78
views
How to cache python pip requirements of docker build progress?
I'm on a very slow internet connection, and the
RUN pip install -r requirements.txt
step of docker compose up --build keeps timing out halfway through.
When I run docker compose up --build again, it ...
- 430
1
vote
0
answers
216
views
How to deploy Celery workers to GCP Cloud Run?
This is my first time attempting to deploy Celery workers to GCP Cloud Run. I have a Django REST API that is deployed as a service to cloud run. For my message broker I'm using RabbitMQ through ...
- 101
2
votes
0
answers
81
views
Query returns None in Celery tasks
I have a celery tasks in my application but when I try to query the database in the functions, some of them returns Does not exist or None
@shared_task
def email_on_assign_reviewer(submission, ...
- 53
0
votes
0
answers
45
views
Django logger only logging to console from celery worker process
I have a following logging config:
"root": {
"level": "ERROR",
"handlers": ["console", "server_file"],
},
"handlers": {
...
- 4,161
0
votes
0
answers
64
views
Celery: How to add a delay to a message sending to SQS
I'm using Celery to consume messages from SQS queue.
The queue is Standard type.
There are cases [exceptions caught] when I explicitly re-enqueue tasks back to the queue.
def run(self):
...
- 4,161
0
votes
0
answers
44
views
Why is my Celery task stuttering when sending frequent progress updates with Django Channels?
I have a loop in a Django application where I process a list of records and send periodic progress updates to the frontend. I'm using async_to_sync from Django Channels to send messages through the ...
- 1
0
votes
0
answers
65
views
Celery task shows success in Flower but is still pending and not executed
I am using Celery with Django. My broker is Redis, and the task results are stored in the database.
I run Celery using the following command:
celery -A core worker -Q celery,send_sms --autoscale=10,0 -...
- 21
2
votes
2
answers
58
views
Django Celery. Publush post on publish_date
I have a Post model. There is a field publish_date. If the post has a planned status, then i need to perform the publishing function on the publish_date. How can I do this?
models.py:
class Post(...
- 121
1
vote
0
answers
32
views
Importing module within the celery task
I wonder whether there could be difference between the following code snippets
from some_module.sub_module import some_function
@app.task(soft_time_limit=16, time_limit=18)
def call_function():
...
- 2,837
0
votes
1
answer
81
views
Using celery-beat as a timer
In my application, I want to run some function once after a certain time interval since the object is created. Is django-celery-beat the right tool for doing this?
0
votes
1
answer
134
views
How to Implement Conditional Task Execution in Celery from API Registration?
I’m building an API that registers tasks to Celery, and I want to implement a conditional workflow based on the results of the tasks. Specifically, I need to execute Task B if Task A returns a certain ...
0
votes
0
answers
19
views
Celery `AsyncResult.state` hangs in `PENDING` (ubuntu)
Considering a task
from celery import Celery
from django.conf import settings
app = Celery(
'api',
broker=settings.CELERY_BROKER_URL,
backend=settings.CELERY_BROKER_URL,
timezone=...
- 5,685
3
votes
1
answer
199
views
Celery tasks run on the wrong worker occasionally
I'm using Django with celery on Heroku with Redis.
I have two queues, called 'celery' and 'letters'
I have two worker types on Heroku
worker (Dyno with lots of RAM). It can run tasks from both queues
...
- 7,341
2
votes
0
answers
108
views
Celery and Redis Command Overload Despite Optimizations in Django App
I’m facing an issue with my Celery + Redis setup for a Django project where the number of Redis commands being executed is skyrocketing despite having made several optimizations. I’m using Upstash ...
- 597
0
votes
1
answer
56
views
Unable to Get Redis Working with Django on Windows
I'm working on a Django Channels project that requires Redis as the channel_layer. However, I've discovered that Redis is not natively supported on Windows, as it's primarily designed for Linux ...
- 53
0
votes
1
answer
94
views
Celery idempotent periodic task - cancel previous instances of task before running the new one
I'm using Celery 5 for task scheduling in my application, with tasks configured to run periodically every 10 minutes. However, I've encountered an issue where, due to server downtime or a backlog in ...
- 3,867
0
votes
1
answer
75
views
flask celery worker not recieving tasks
I'm working on a task where I need to schedule my application to execute at regular intervals using Celery Beat schedule.
The main issue I'm facing is that Celery is not triggering my application at ...
1
vote
0
answers
41
views
raised unexpected: TypeError('send_verification() takes 1 positional argument but 3 were given') in celery
class CreateAccountView(APIView):
permission_classes = []
def post(self, request):
print(request.data)
email = request.data.get('email')
if User.objects.filter(email=...
- 11
0
votes
0
answers
39
views
Why celery took GBs to store chord task result
I am running one simple celery task
task_list = [a.s(id) for id in id_list]
a_callback = callback_function.s()
chord(task_list)(a_callback)
so when the task list is lets say 1-2L items, the table ...
0
votes
0
answers
167
views
How to handle large task_kwargs in Celery to avoid ellipsis in TaskResult
I am using Celery for asynchronous task processing in my Django application. Some of my tasks involve handling large amounts of data passed as task_kwargs. However, I have noticed that when the ...
- 334
0
votes
1
answer
56
views
Celery, Redis: multiplicating tasks and not executing them
Celery eats up RAM.
We're using Celery in Django REST with Redis as the broker. Celery is used to send and retry sending callbacks, if unsuccessful (retry policy was to try sending callbacks with ...
- 173
1
vote
0
answers
185
views
How to change Celery worker concurrency in Django app
Recently I noticed that celery eats up a lot of memory in the server. One instance eating like 10% of mem
in processed I see quite default situation:
/usr/local/bin/python -m celery -A apps worker -l ...
- 399
1
vote
0
answers
56
views
Scrapy with Celery
How to run spiders sequentially (one after another) with a 2-minute interval in my task?
signals.py
@receiver(post_save, sender=ParseCategoryUrl)
def start_parse_from_category_url(sender, created, ...
- 93
0
votes
1
answer
625
views
How to properly dockerize Celery with Django?
I am having a little trouble dockerizing Celery alongside my Django project. Recently I had the need to add Celery to my projects, and I have worked with it prior, but not in a Docker environment.
I ...
- 90
1
vote
1
answer
29
views
Not able to read length of queue from code in Django Gunicorn app but works fine in console
The following code of get_queue_length gives queue length as zero all the time but when I run the same set of command in Django shell it works fine and gives total number of object in the queue
import ...
1
vote
0
answers
75
views
Celery chord running a group of chains hangs
I am using a similar orchestration pattern as what's described in this Stackoverflow question: celery chord with group of chains with exception in chain
The key orchestration code is this, which I've ...
- 4,718
0
votes
1
answer
26
views
Celery name task execution and get task execution by name
I have a Celery task called my_task. I create multiple exeucutions of the task by calling my_task.delay() multiple times. I want to give a unique label/name to each execution, and to be able to get ...
- 1,112
0
votes
1
answer
133
views
How can celery use Django ORM?
How can celery use Django ORM?
Celery and django are diffent processes.
But, Celery worker can use Django Application code.(ORM, Model..)
I think celery process have django application code. Is it ...
- 3
0
votes
0
answers
220
views
Docker / Celery / Django : Run a task with `celery` in a container while my `django` server is running in another container. It's possible?
I'm working on a project that uses Django, I'm a beginner with Django and Docker.
For a feature, I decided to create a task with Celery (the feature consist to create a matchmaking queue)
But, I use ...
- 5
1
vote
0
answers
116
views
How to handle memory intensive task causing WorkerLostError with Celery and HuggingFaceEmbedding?
I'm trying to use celery to handle the heavy task of creating a new qdrant collection every time a new model is created, I need to extract the content of the file, create embedding and store it in ...
- 56
0
votes
1
answer
294
views
How to get realtime stdout from a celery task?
I have a Django project in which i'm running some background tasks with celery, one of my goal is to show real-time stdout of the running task so that the user can see the progress. the issue is that ...
1
vote
0
answers
66
views
How can I ensure synchronized incrementation of two counters in a Celery async task
I'm encountering a scenario where a single endpoint is receiving multiple concurrent requests for image uploads. Upon successful upload, two counters need to be incremented synchronously. Initially, I ...
0
votes
1
answer
98
views
Django-celery-beat send task, worker receive it and don't execute
That's my libs:
celery==5.3.6
django-celery-beat==2.5.0
django-celery-results==2.5.1
redis==5.0.1
settings.py:
INSTALLED_APPS = [
"django.contrib.admin",
"django.contrib.auth&...
- 9
0
votes
2
answers
211
views
Celery and Celery beat wit django after running in docker container running some job
I am trying to run django celery and celery beat.
After I start celery and celery beat
this process is running every second. Is this normal?
celery-1 | [2024-03-10 16:08:11,479: INFO/MainProcess]...
- 153
0
votes
1
answer
329
views
Storing files to S3 not working inside Celery worker
I am facing issues with storing files in S3 from django's celery task. I am doing following things
Get data from DB
Create the XLSX file with BytesIO
Storing that file with file.save method of Django
...
- 60
0
votes
0
answers
155
views
Celery & Django - does the main process wait for tasks to finish?
I've just started work on a project that uses Django & Celery. I'm quite new to both. One of the endpoints has code that looks like this:
task_do_something.delay()
return Response(
{&...
1
vote
1
answer
473
views
How to manage the increase in memory consumption of redis-server in a Docker environment?
I am using the redis-server as part of a Docker stack in a Django project that uses Celery Beat for scheduled tasks. While monitoring the processes with the htop command, I noticed that the memory ...
- 93
0
votes
0
answers
78
views
Django Celery not able to fetch data from particular DB schema
We are facing the issue fetching the data from particular schema's table which is available and the data fetching issue occurs intermediately after few hours of restart of supervisorctl where celery ...
0
votes
1
answer
1k
views
How to set a (Django) celery beat cron schedule from string value
I am using Django with celery beat. I would like to configure the cron schedule via env var (string value of the cron).
We are currently setting the cron schedule like this, using celery.schedules....
- 185
0
votes
1
answer
2k
views
Celery doesn't detect and register tasks in django python
I'm facing a problem where I can't set up periodic tasks in my application in django framework using Celery.
I've literally tried everything i could find adressing this issue on the internet, ...
- 11
1
vote
1
answer
1k
views
Combining Celery Gevent Pool with asyncio (asgiref)
I have lots of async code which is used for websocket connectons. Idea is simple: fetch data from websocket, calculate some math and do lots of io.
data = await websocket.recv()
# Do some math
...
- 67
3
votes
1
answer
683
views
Connecting to Amazon SQS key (SSE-SQS) Encrypted SQS Queue via Django Celery [sqs]
I am unable to get my Celery working running while connecting to an SQS Encrypted Queue
This is my settings.py for my django project
SQS_AWS_ACCESS_KEY_ID = 'xxxx'
SQS_AWS_SECRET_ACCESS_KEY = 'xxxx'
...
0
votes
0
answers
122
views
django celery tasks are executed before establishing a websocket connection
The user uploads the file to the form, after which the form with the file is sent to the server, and upon successful validation, I perform the following actions:
def form_valid(self, form):
...
- 49