Force Celery to run next task in queue?

Question

Is there any way to make Celery recheck if there are any tasks in the main queue ready to be started? Will the remote command add_consumer() do the job?

The reason: I am running several concurrent tasks, which spawn multiple sub-processes. When the tasks are done, the sub-processes sometimes take a few seconds to finish, so because the concurrency limit is maxed out by sub-processes, a new task from the queue is never started. And because Celery does not check again when the sub-processes finish, the queue gets stalled with no active tasks. Therefore I want to add a periodical task that tells Celery to recheck the queue and and start the next task. How do I tell Celery this?

Casey Falk · Accepted Answer · 2014-08-08 15:26:01Z

0

From the docs:

The add_consumer control command will tell one or more workers to start consuming from a queue. This operation is idempotent.

Yes, add_consumer does what you want. You could also combine that with a periodic task to "recheck the queue and start the next task" every so often (depending on your need)

edited Aug 8, 2014 at 15:26

answered Jul 2, 2014 at 17:13

Casey Falk

2,6771 gold badge21 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

dwitvliet Over a year ago

No downvotes. I just removed my upvote, as you don't actually answer my problem, but just confirms my suspicion that "it looks like add_consumer does" the job.

Casey Falk Over a year ago

Fair enough, but how does this not answer your question? Your question as-written is: "will the remote command add_consumer() do the job?" The answer is "yes." Please clarify. :)

dwitvliet Over a year ago

I need a definite answer from someone who actually tried and knows. The docs are vague, which is why I asked in the first place. "start consuming from a queue" can mean multiple things when that queue is already being consumed from, but it is just on pause because the concurrency limit has been reached.

Collectives™ on Stack Overflow

Force Celery to run next task in queue?

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related