Retry Celery tasks with exponential back off

Question

For a task like this:

from celery.decorators import task

@task()
def add(x, y):
    if not x or not y:
        raise Exception("test error")
    return self.wait_until_server_responds(

if it throws an exception and I want to retry it from the daemon side, how can apply an exponential back off algorithm, i.e. after 2^2, 2^3,2^4 etc seconds?

Also is the retry maintained from the server side, such that if the worker happens to get killed then next worker that spawns will take the retry task?

fjsj · Accepted Answer · 2021-12-14 11:17:21Z

164

The task.request.retries attribute contains the number of tries so far, so you can use this to implement exponential back-off:

from celery.task import task

@task(bind=True, max_retries=3)
def update_status(self, auth, status):
    try:
        Twitter(auth).update_status(status)
    except Twitter.WhaleFail as exc:
        raise self.retry(exc=exc, countdown=2 ** self.request.retries)

To prevent a Thundering Herd Problem, you may consider adding a random jitter to your exponential backoff:

import random
self.retry(exc=exc, countdown=int(random.uniform(2, 4) ** self.request.retries))

edited Dec 14, 2021 at 11:17

fjsj

11.3k11 gold badges43 silver badges61 bronze badges

answered Mar 17, 2012 at 18:54

asksol

19.6k5 gold badges67 silver badges69 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Quintin Par Over a year ago

Do you know if this is a server side retry or the client is kept to wait? If the client is kept to wait then it’s bad.

idanzalz Over a year ago

As far as I know the countdown attribute sets an eta for the task at the MQ backend (e.g. RabbitMQ). So it is not set on the client side.

asksol Over a year ago

client is not kept to wait unless you do result.get() which is an explicit request to wait for the result to be ready, but there's also a timeout argument and there's a RETRY state so you can check if the task is being retried (and what the reason for the retry was)

robbyt Over a year ago

For celery 3.1, you should use @task(bind=True) and celery will pass self into the function as the first argument, so you would change the args to be def update_status(self, auth, status): which then gives you access to self.retries

tutuDajuju Over a year ago

thanks @robbyt ! just a small correction - retries is an attribute of request, so self.request.retries is the proper call.

|

Sebastian Dix · Accepted Answer · 2023-11-28 22:03:13Z

61

As of Celery 4.2 you can configure your tasks to use an exponential backoff automatically: https://docs.celeryq.dev/en/stable/userguide/tasks.html#automatic-retry-for-known-exceptions

@app.task(autoretry_for=(Exception,), retry_backoff=2)
def add(x, y):
    ...

(This was already in the docs for Celery 4.1 but actually wasn't released then, see merge request)

edited Nov 28, 2023 at 22:03

Sebastian Dix

256 bronze badges

answered Sep 28, 2017 at 11:19

Rupert Angermeier

1,1959 silver badges15 bronze badges

5 Comments

kororo Over a year ago

Nice catch, scratching my heads in 4.1.0, why my parameter of "retry_backoff" not respected.

rdrey Over a year ago

@kororo it doesn't seem to work with self.retry, only other exception types

qwertysmack Over a year ago

With this approach you also benefit from the built in retry_jitter (defaulted to True) which avoids the Thundering Herd Problem mentioned in asksol's answer

jorf.brunning Over a year ago

This is the correct answer given that it is built-in, and does not require manually handling countdown

Sarang Over a year ago

Does this also work when retry() is called? It doesn't seem to work for non-automatic retries (on Celery 4.2.2 at least). Anyone has any idea?

boatcoder · Accepted Answer · 2022-10-23 12:35:45Z

3

FYI, celery has a util function to calculate exponential backoff time with jitter here, so you don't need to write your own.

def get_exponential_backoff_interval(
    factor,
    retries,
    maximum,
    full_jitter=False
):
    """Calculate the exponential backoff wait time."""
    # Will be zero if factor equals 0
    countdown = min(maximum, factor * (2 ** retries))
    # Full jitter according to
    # https://www.awsarchitectureblog.com/2015/03/backoff.html
    if full_jitter:
        countdown = random.randrange(countdown + 1)
    # Adjust according to maximum wait time and account for negative values.
    return max(0, countdown)

edited Oct 23, 2022 at 12:35

boatcoder

18.2k21 gold badges126 silver badges188 bronze badges

answered Oct 29, 2021 at 7:10

lgylym

2061 silver badge8 bronze badges

1 Comment

dKen Over a year ago

In the future, avoid link-only answers, as links tend to go stale over time. Best to also include a code snippit and explanation in your answer for maximum upvotes and value-add. Edit: case in point, this answer's link is already broken stackoverflow.com/a/46467851/366529

Collectives™ on Stack Overflow

Retry Celery tasks with exponential back off

3 Answers 3

7 Comments

5 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

7 Comments

5 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related