Celery + Python: Queue time consuming tasks within another task

Question

I want to query an api (which is time consuming) with lots of items (~100) but not all at once. Instead I want a little delay between the queries.

What I currently have is a task that gets executed asynchronously and iterates over the queries and after each iteration waits some time:

@shared_task
def query_api_multiple(values):
    delay_between_queries = 1

    query_results = []

    for value in values:
        time.sleep(delay_between_queries)

        response = query_api(value)
        if response['result']:
            query_results.append(response)

    return query_results

My question is, when multiple of those requests come in, will the second request gets executed after the first is finished or while the first is still running? And when they are not getting executed at the same time, how can I achieve this?

You should use ETA – Scheduling a task for later, for doing that asynchronously using ETA(estimated time of arrival) like: later = now + timedelta(hours=1) access_awful_system.apply_async((object_id), eta=later) reference ETA. — Fady Saad
– Fady Saad, Commented Sep 3, 2017 at 22:29

skovorodkin · Accepted Answer · 2017-09-03 22:31:37Z

2

You should not use time.sleep but rate limit your task instead:

Task.rate_limit

Set the rate limit for this task type (limits the number of tasks that can be run in a given time frame).

The rate limits can be specified in seconds, minutes or hours by appending “/s”, “/m” or “/h” to the value. Tasks will be evenly distributed over the specified time frame.

Example: “100/m” (hundred tasks a minute). This will enforce a minimum delay of 600ms between starting two tasks on the same worker instance.

So if you want to limit it to 1 query per second, try this:

@shared_task(rate_limit='1/s')
def query_api_multiple(values):
    ...

edited Sep 3, 2017 at 22:31

answered Sep 3, 2017 at 22:23

skovorodkin

10.5k2 gold badges41 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

PirateApp Over a year ago

This rate limit only applies per worker, either they are asking you to add this task to a queue and then have only 1 worker process that queue or not add a rate limit at all, is there a better way of handling this where all workers would adhere to the rate limit

Sean Azlin · Accepted Answer · 2017-09-05 02:55:17Z

1

Yes, if you create multiple tasks then they may run at the same time.

You can rate limit on a task type basis with celery if you want to limit the number of tasks that run per period of time. Alternatively, you could implement a rate limiting pattern using something like redis, combined with celery retries, if you need more flexibility than what celery provides OOtB.

answered Sep 5, 2017 at 2:55

Sean Azlin

9267 silver badges22 bronze badges

Collectives™ on Stack Overflow

Celery + Python: Queue time consuming tasks within another task

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related