Celery check pending tasks number before specified taskid

Question

Does celery support returning pending tasks number before given task id?

For example, without celery worker started, I push task1, task2, task3, all the three are pending, now, what I wanna is, if I give task3, it tells me there are 2 pending tasks before 3.

I use celery celery 4.1, rabbitmq 3.5.4 as broker, and redis 3.2.9 as result backend.

Although I can get rabbit queue depth by management API(e.g. get_queue_depth from pyrabbit package), this results the whole queue depth, not pending number before specified task id.

And I know I could maintain a queue managing pushed task ids by myself.

But I wanna if there is any easy way by celery or rabbitmq itself.

Thanks.

Got the same problem.

Wesley
– Wesley

2017-09-19 10:03:32 +00:00
Commented Sep 19, 2017 at 10:03 — Wesley
– Wesley, Commented Sep 19, 2017 at 10:03

ItayB · Accepted Answer · 2017-09-19 20:15:28Z

1

I'm not sure if it answer your question but there is control client that can help you to inspect reserved tasks, active tasks and so on..

i = app.control.inspect()
i.reserved()

#output:
[{'worker1.example.com':
    [{'name': 'tasks.sleeptask',
      'id': '32666e9b-809c-41fa-8e93-5ae0c80afbbf',
      'args': '(8,)',
      'kwargs': '{}'}]}]

for more info: http://docs.celeryproject.org/en/latest/userguide/workers.html#dump-of-reserved-tasks

You can also monitor/inspect from command line: http://docs.celeryproject.org/en/latest/userguide/monitoring.html#commands

answered Sep 19, 2017 at 20:15

ItayB

11.6k11 gold badges61 silver badges94 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Wesley Over a year ago

Although tasks in i.reserved() is list, but sequence is not exactly as the order be executed. So, this shouldn't work.

ItayB Over a year ago

@Wesley did u try?

Wesley Over a year ago

not yet... I think you wanna disable message prefetching, I need to find way to close this guy, seems set CELERYD_PREFETCH_MULTIPLIER = 1 is not enough, but first comes a problem that, set this guy to 1 will decrease performance. Don't know if there is any other way not decreasing performance

ItayB Over a year ago

I'm working with celery in high load and set this one (for using priority queues) - it shouldn't be a problem. BTW you can still increase the worker concurrency..

Wesley Over a year ago

stackoverflow.com/questions/16040039/… this post explain a lot about prefetching. However, we make a fatal mistake till now...that is, inspect is only for running celery workers, not for rabbitmq. We cannot get the pending tasks within rabbitmq from celery inspect. I think we have to solve the issue from the mq side. I am thinking about get_messages of pyrabbit package.

|

Collectives™ on Stack Overflow

Celery check pending tasks number before specified taskid

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related