Same Message Group Tasks executing in parallel in Celery with Amazon SQS FIFO

Question

I have a setup where I'm using Celery as the task queue with Amazon SQS FIFO. My goal is to ensure sequential processing of tasks within the same message group ID, while allowing tasks with different message group IDs to be processed in parallel. However, despite following the recommended configurations and understanding the behavior of SQS message groups, I'm experiencing parallel processing of tasks within the same message group by multiple Celery worker processes. How can I ensure that tasks with the same message group ID are processed sequentially by a single worker process, while maintaining parallel processing for tasks with different message group IDs?

Some extra details (for reference) : For celery I haven't used --concurrency setting, so it is by default spawning 4 pool processes (no. of cores). I am passing message group id using following syntax :

    message_properties = {
        "MessageGroupId": f"{supplier_id}"
    }
    celery_task.s(
        param1, **message_properties
    ).apply_async(**message_properties)

and I have made sure the queue is fifo and it ends with .fifo . additional settings - { 'polling_interval': 60, 'wait_time_seconds': 10, 'visibility_timeout': 600 }

I tried to play around with the settings but nothing seems to work. Even with the same group ids the task are still being processed in parallel by spawned processes of celery worker.

Madiha Khalid · Accepted Answer · 2023-07-07 18:08:34Z

0

I am not sure exactly but you can try this celery configuration prefetch_multiplier=1 Check this documentation

The prefetch limit is a limit for the number of tasks (messages) a worker can reserve for itself.

And you can also set the concurrency=1

With these both setting each worker will process one request at a time per Group ID hence we could achieve the sequential message processing.

answered Jul 7, 2023 at 18:08

Madiha Khalid

3904 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Manish Patel Over a year ago

yes but it take a hit on performance. By using amazon sqs fifo , I wanted to run the other tasks parallely but only the tasks within the same message group to be executed sequentially which is not happening. Setting concurrency to 1 may work as workaround but I don't see it as a solution.

Manish Patel Over a year ago

@GiovaniBarcelos No, I gave up in the end and went with creating a new queue for the sequential tasks setting its concurrency to 1. If you do find a solution however, please do post it here.

Collectives™ on Stack Overflow

Same Message Group Tasks executing in parallel in Celery with Amazon SQS FIFO

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related