Celery multiple task execution due to prefetching

Question

It seems that Celery (v4.1) can be either used with some prefetching of tasks, or with CELERY_ACKS_LATE=True (Discussed here)
We currently work with CELERY_ACKS_LATE=False and CELERYD_PREFETCH_MULTIPLIER=1

In both cases, there are unacknowledged messages in Rabbit.
At times we suffer from network issues that cause Celery to lose the connection to Rabbit for few seconds, getting these warnings: consumer: Connection to broker lost. Trying to re-establish the connection...
When this happens, the unacknowledged messages, turn back to Ready, what seems to be the standard behaviour, and are being consumed by another consumer.
This causes a multiple execution of the tasks, as the consumer started a prefetched task in the worker process, but couldn't ack it to Rabbit.

As it seems that its impossible to guarantee that tasks will get executed exactly once in Celery without external tools, how is it possible to ensure that tasks are executed at most once?

----- Edit ----

One approach I'm considering is to use the task's self.request.delivery_info['redelivered'] and fail tasks that were redelivered.
While achieving the goal of "executing at most once" this will have a high rate of false positives (tasks that weren't already executed)

docs.celeryproject.org/en/latest/tutorials/task-cookbook.html — bruno desthuilliers
– bruno desthuilliers, Commented Jan 12, 2018 at 12:18
Thanks @brunodesthuilliers , I'm aware of solutions using external tools like you've offered but I'm looking for something using Celery only. I've also changed the problem description as I'm OK with ensuing "at most once" guarantee if the number of tasks executing 0 times is low — dimamah
– dimamah, Commented Jan 16, 2018 at 7:10

amirouche · Accepted Answer · 2018-01-16 10:12:29Z

1

There is difference between the task is executed once and there is extra side effect when the task is executed multiple times ie. if your tasks are not idempotent then executing twice the same task will lead to bugs.

What I recommend is to allow task to execute several times, but making them idempotent so they have no effect if they were already executed.

answered Jan 16, 2018 at 10:12

amirouche

7,9417 gold badges42 silver badges100 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

dimamah Over a year ago

I know that this is indeed the prefered approach but unfortunately my task are not idempotent and the effort to make them idempotent is huge

amirouche Over a year ago

Then your question, doesn't have enough details.

Collectives™ on Stack Overflow

Celery multiple task execution due to prefetching

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related