Message Queue VS Task Queue architecture

Question

I want to build a product that can perform some Internet scans (in Python) to collect various kinds of data.

I want to design it with tasks that perform these collecting jobs.

There can be multiple scans that run parallel on different inputs, so tasks can be duplicated since they have different inputs to operate on.

I wonder which architecture would fit for it, and what technologies are the best.

I thought of using RabbitMQ to store the tasks and Redis to store inputs.

The initial inputs trigger the scan, then each task spits its output that might be the input for other tasks.

What do you think of this possible design? Can it be improved? Other technologies?

During the system design you should try to avoid using specific technologies. Rather than you should rely on abstract components (like message-queue, distributed cache, etc.). Drawing diagrams to depict the data/communication flow would also help you (and SO readers) to better understand your problem domain and your proposed solution. — Peter Csala
– Peter Csala, Commented Feb 2, 2023 at 11:58

danronmoon · Accepted Answer · 2024-04-02 14:52:09Z

1

It depends on the size of the inputs. If those are relatively small I would go with just message broker and sending everything in the message (i.e. the task type and its inputs) - otherwise some outside store is better used. Depending on the durability requirements possibly a persistent storage (like database) should be considered.

edited Apr 2, 2024 at 14:52

danronmoon

3,8735 gold badges36 silver badges58 bronze badges

answered Feb 1, 2023 at 16:47

Guru Stron

151k11 gold badges186 silver badges232 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Maxim Fateev · Accepted Answer · 2023-02-05 00:53:26Z

0

One option is to use an existing orchestrator which hides most of the complexity instead of crafting a custom solution based on queues and storage. Look at temporal.io open source project which allows orchestrating tasks using high level programming language.

answered Feb 5, 2023 at 0:53

Maxim Fateev

6,9603 gold badges25 silver badges40 bronze badges

Collectives™ on Stack Overflow

Message Queue VS Task Queue architecture

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related