Change default_retry_delay for a celery task at runtime?

Question

Is there a way to modify the retry delay for celery tasks at runtime? Or is there a global config value that can be changed to override the 180s default value?

I have set up tasks with exponential back-off (as described here: Retry Celery tasks with exponential back off) but I want to override this value when integration testing.

The reason is that I often end up triggering the 180s default value if exceptions are raised within an exception handler, which seems to bypass and ignore the countdown argument.

class BaseTask(celery.Task):
    def on_retry(self, exc, task_id, args, kwargs, einfo):
        """Log the exceptions at retry."""
        logger.exception(exc)
        logger.warning('Retry: {}.'.format(self.request))
        super().on_retry(exc, task_id, args, kwargs, einfo)

    def on_failure(self, exc, task_id, args, kwargs, einfo):
        """Log the exceptions on failure."""
        logger.exception(exc)
        logger.error('Failure: {}.'.format(self.request))
        super().on_failure(exc, task_id, args, kwargs, einfo)

    @property
    def backoff_countdown(self):
        return int(random.uniform(2, 4) ** self.request.retries)

@celery.task(bind=True, base=BaseTask)
def process(self, data):
    try:
        return some_task(data)
    except Exception as exc:
        raise self.retry(exc=exc, coundown=self.backoff_countdown)

Regardless of what I set for self.backoff_countdown (even just returning 1) I end up with tasks being retried in 180s, which makes it really hard to run integration tests with reasonable timeouts.

kkiat · Accepted Answer · 2016-11-30 10:33:48Z

1

Refer to document http://docs.celeryproject.org/en/latest/userguide/tasks.html#using-a-custom-retry-delay, can set default_retry_delay or set the countdown value.

@celery.task(bind=True, default_retry_delay=30 * 60) # retry in 30 minutes.
def process(self, data):
    try:
        return some_task(data)
    except Exception as exc:
        # Retry in 5 minutes
        raise self.retry(exc=exc, countdown=5 * 60)

answered Nov 30, 2016 at 10:33

kkiat

5713 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

sas Over a year ago

Yes, I know of that setting but it's not something I can change at runtime - at least not that I can see. Maybe I'm phrasing this badly. I'm ok with the 3min normal retry delay and I don't want to change that in the source. However, I do want to override that value when I run this same source in a docker image by some configuration setting that I can inject or by some instrumentation that I add to my tests.

kkiat Over a year ago

How about override the retry method of Task object?

sas Over a year ago

This code is running in a docker container. The best I can do is inject config files or set env variables. I guess I could try to inject a value for @celery.task(... default_retry_delay=...) via an env variable but that seems awfully convoluted and is not something I'd like to ship.

kkiat Over a year ago

From your code, i don't see your method is override the default_retry_delay value, you can try to override the retry method of Task object with your countdown parameter.

sas Over a year ago

I think a better description of my problem would be: Given the tasks are defined as shown and assuming that I cannot change them, how could I adjust default_retry_delay from the 'outside' to achieve the same effect as via the @celery.task decorator.

Collectives™ on Stack Overflow

Change default_retry_delay for a celery task at runtime?

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related