-1

Celery seems to be picking both my old code and new code. I have tried clearing cache, clearing the broker queue(redis), restarting celery etc. But none of them seem to be fixing this issue.

For context, we have new releases going to various servers periodically. The web application uses Django rest framework in the backend and celery for scheduling asynchronous tasks. Recently when we deployed new version of the code, the application was behaving very strangely. It had artifacts from the old code that was being run and parts of the new code as well. This was very weird behaviour until we found a github thread(https://github.com/celery/celery/issues/3519) which outlines the issue exactly as we had faced. There seems to be no good answers for this issue in that thread, so posting it here so that if anyone with celery knowledge knows of a workaround where we can stop celery from picking up the old artifacts.

The deployment is done through Jenkins build scripts. Please find below the script for the same. For obvious reasons, I have replaced our application name with "proj".

sudo /bin/systemctl stop httpd
sudo /bin/systemctl stop celery
/bin/redis-cli flushall
/srv/proj/bin/pip install --no-cache-dir --upgrade -r /srv/proj/requirements/staging.txt
/usr/bin/git fetch
/usr/bin/git fetch --tags
/usr/bin/git checkout $TAG
/srv/proj/bin/python manage.py migrate
sudo /bin/systemctl restart httpd
sudo /bin/systemctl restart proj
sudo /bin/systemctl start celery
Improve this question
8
  • Either describe the issue in detail here if you hope to find a solution through Stack Overflow, or engage in the ticket you linked if you hope to get the maintainers to fix it. Commented May 26, 2022 at 6:45
  • @deceze I have already described the issue. "Celery seems to be picking both my old code and new code". I was just hoping if anyone here knows of a workaround to make celery "stop" picking up my old code. What other details do you need? Commented May 26, 2022 at 6:48
  • A minimal reproducible example, please. Commented May 26, 2022 at 6:56
  • This is not an issue that can be reproduced:) This is a behaviour that happens very randomly and rarely. People who have worked with celery for a long time might be aware of this. Commented May 26, 2022 at 7:02
  • And for those who aren’t, at least a more detailed explanation will be necessary. Commented May 26, 2022 at 7:07

1 Answer 1

Reset to default
2

OP, the problem is almost invariably that your servers include old code on them. There are two possible issues:

  1. the checkout command fails. checkout can fail to switch branches if there are local changes on a directory. In addition, the deployment script doesn't pull the latest changes to the server. The more common approach that we use is to have the venv in another directory, and then on deploy clone the repository to a new directory (versioned). Then switch the "main" app directory link to point to the latest version. The latter, for example, is the approach used in aws's elastic beanstalk.
  2. there are stray processes of celery still running. Depending on how you started / stopped celery, there could be stray processes of celery still hanging around and running your old code.

Ultimately, you won't be able to diagnose this problem without confirming that all of your servers are 100% identical. So, if you have devs or admins that ssh to these boxes, chances are that one or more of them made some change that affected (1) or (2).

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

We finally diagnosed the problem. It was a stray process of celery that was still running. The regular celery stop/purge command was clearly not working. We had to do a pkill on all the celery processes. Hope Celery fixes this in future versions. Thanks much for your answer.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.