3,987 questions
0
votes
0
answers
14
views
Guidewire triggering a custom event in messaging queue
Im trying to create a new message queue which would send a message to external system but im not able to trigger the new custom event. From the docs ive seen we need to use .addevent() method but ...
0
votes
0
answers
37
views
Message queue where each key is processed by only one consumer at a time, but other messages for the same key can be assigned to other consumers
I'm building a distributed system that processes messages from hundreds of thousands of sensors, each identified by a sensor_id.
I need to guarantee:
Strict ordering per sensor_id
No two consumers ...
- 1,510
-3
votes
0
answers
51
views
How to paginate a chat conversation list (1,000+ threads) so new messages move the thread to the top without breaking pagination?
I’m building a chat app using Node.js, Express, MongoDB, and Socket.io. Right now, I load all conversations (around 1,000+) at once, and it’s making the app slow.
I want to add pagination to the ...
-1
votes
2
answers
70
views
Why retry cound does not increase upon exception thrown?
In Symfony I use a custom Serializer upon a transport:
framework:
messenger:
transports:
async: '%env(MESSENGER_TRANSPORT_DSN)%'
failed: 'doctrine://default?...
- 10.4k
2
votes
2
answers
75
views
How to use multiprocessing module with interdependent tasks?
The existing examples of the multiprocessing-module assume, the tasks are independent, but my situation is "hairier".
I need to process several hundred thousands of widgets. These are ...
- 4,266
0
votes
0
answers
107
views
Azure Functions (Flex Consumption) – Storage Queue trigger not firing when message added
I’m working on an Azure Functions project using the new Flex Consumption model with @azure/functions in TypeScript.
I have two functions:
Producer – sends a message to a storage queue:
export async ...
- 13
0
votes
1
answer
117
views
MassTransit scheduled redelivery with Kafka and Hangfire does not work
I’m trying to configure scheduled redelivery in MassTransit when using Kafka as the transport.
Immediate retries work fine using UseMessageRetry.
Since Kafka does not support delayed redelivery, I’m ...
- 45
0
votes
0
answers
99
views
Win32 Mouse Click is observed through Message Queue, but not working
I am making a Win32 app. In this, there are two windows - the top one is transparent, and the bottom one is normal. I am trying to achieve this effect:
When the click happens on the top transparent ...
- 5,310
0
votes
1
answer
60
views
NestJS @EventPattern with RMQ topic exchange does not create binding (no bindingKeys, custom exchange)
I’ve created a service in NestJS and connected it to RabbitMQ.
The connection is established successfully, but I don’t receive any messages unless I manually add a binding in the RabbitMQ Management ...
- 19
0
votes
1
answer
151
views
How to use RabbitMQ Publisher Confirms and Returns for Reliable Message Delivery
I'm using RabbitMQ with Spring AMQP and want to ensure reliable message delivery from the producer to the RabbitMQ broker. To achieve this, I've enabled Publisher Confirms and Publisher Returns.
...
- 23
0
votes
0
answers
37
views
Parallel sagas listening to the same channel are encountering a race condition
I'm attempting to develop an understanding of the Saga and Outbox patterns, and am working on development now. I'm using RabbitMQ and NestJS. I've got three services, Orders, Inventory, and Shipping. ...
- 73
0
votes
0
answers
133
views
BullMQ: Flow parent job's 'failed' event not triggered when child jobs fail
I'm implementing a job flow using BullMQ (latest version) with NestJS, and I've encountered an issue with error handling in flow jobs.
The setup
I have a parent flow job and multiple child jobs. The ...
- 1,445
1
vote
0
answers
63
views
How to perform Geo-Replication in Apache Pulsar without creating duplicate messages?
I’m designing a multi-datacenter architecture using Apache Pulsar with geo-replication enabled.
Architecture Overview:
Apache Pulsar version: 4.0.2
Helm Chart version: pulsar-3.9.0
BookKeeper: 5 ...
0
votes
0
answers
25
views
Asynch integration approach between two ERP systems
We need to integrate two ERP systems which have to exchange many Entities (Items, Vendors, Prices ect.) and business Documents (Purchase orders, Sales orders, Warehouse transfer orders, ect.).
The ...
- 1
0
votes
1
answer
99
views
Understanding GCP Pub/Sub Sent Metric and Ack Metric With Dead Lettering
I am testing my Pub/Sub setup with dead lettering enabled. Inside my subscriber application, I have not added any ack logic so all messages will not be acked by my subscriber. I set the maximum ...
0
votes
2
answers
282
views
Discord Bot and a website communication, what should be the system design
I'm building a website that lets users add new items in their profiles. Users can also do the same using Discord bot's commands. In both cases the Discord bot sends a message New item added by @user ...
- 5,239
0
votes
0
answers
39
views
How to Handle APIs which takes more than 5 seconds in serverless applications
In One of the requriement we are hitting a third Party API which does some calculation at the backend and dependent on the various factors it can take time upto 5 seconds.
Usually serverless is to be ...
- 29
0
votes
1
answer
38
views
Send saga is not available in IoC anymore
I have a problem with Masstransit and the Saga feature.
I cannot explain the following errors:
Error: MassTransit.ReceiveTransport[0]
R-FAULT rabbitmq://localhost/monitoring-job-saga 489a0000-...
- 1
0
votes
0
answers
43
views
How to dynamically extend the holding time of a message for the message consumer in RabbitMQ?
Simply put, I want to implement the same function of ServiceBusReceiver.RenewMessageLockAsync of Azure Service Bus queue in RabbitMQ. This function lets the message consumer "extend" the &...
- 2,297
1
vote
0
answers
400
views
BullMQ Worker with rate limits and unique group processing
I’m working with BullMQ (v5+) and need to implement a worker to process messages for different chats, with the following requirements:
Rate Limit:
Each queue should have a rate limit of 1 job per 3 ...
- 41
1
vote
0
answers
163
views
Does SQS FIFO have any Issues with many MessageGroupIds?
I'm fairly new to working with FIFO SQS, and a question has emerged. Does using many different MessageGroupIds have any performance downsides or raise any concerns?
I'm working with an order queue and ...
0
votes
1
answer
394
views
When are Azure Storage Account Poison Queues Created?
We have Terraform that deploys our Azure Storage Accounts and associated Storage Account Queues.
I notice that there are also associated 'queuename-poison' queues when I view the deployed ...
- 30.6k
0
votes
0
answers
47
views
Can invariants span across multiple aggregate roots?
I've got a bit of a theoretical question surrounding DDD and aggregates. It really boils down to this: Yes, we all know that within an aggregate everything must stay consistent. But does that mean ...
- 73
0
votes
0
answers
94
views
Message Queue causing core dump when checking is message available
When I'm trying to check if new message is available in message queue, I'm getting core dump error.
I'm calling isDataAvailableInReceiveDataQueue method to check if data is available in message queue.
...
- 2,041
0
votes
0
answers
47
views
Hangfire: How to tune degree of Concurrency depending on Parameters?
Using Hangfire, we'd like to fine-tune the degree of concurrency depending on parameter values.
Given many buckets, each with the same multiple segments
Bucket 1
Segment A
Item 1A-1
Item 1A-2
...
- 1
1
vote
0
answers
40
views
Spring boot app cannot connect with Kafka within the container
In short
To repeat the issue, you may clone the project from:
https://github.com/ethanalef/event-driven-model
Then build both listener and publisher projects with maven, build the image with ...
- 75
0
votes
0
answers
36
views
Sending different stomp message types to the same queue
I'm writing a Spring Boot application (App) which handles STOMP messages sent from a web browser (or Agent). The messages (Commands) arrive, are repackaged and published to a topic which a second ...
- 11k
1
vote
2
answers
269
views
How to solve Boost Error:'boost::interprocess_exception::library_error'
I created a message_server in the qnx environment, and the "library_error" always appeared :
#include <boost/interprocess/ipc/message_queue.hpp>
int main() {
boost::interprocess::...
- 11
0
votes
1
answer
349
views
How to Invoke queue trigger function?
I have created a simple trigger function from a template, this is working perfectly fine if I debug from Visual Studio, which is
once I add message manually to my queue, I can see that message in ...
- 49
2
votes
0
answers
99
views
How to efficiently implement auditing for high-throughput background service in .NET 8? [closed]
I have a .NET 8 background service responsible for two primary tasks:
1. Push JSON data packets to message brokers (ActiveMQ, RabbitMQ, or JMS queues):
Sends approximately 25,000 JSON data packets ...
- 6,105
0
votes
0
answers
58
views
Why are messages lost after consumer recovery in RabbitMQ/MassTransit publisher-consumer setup?
In my MassTransit + RabbitMQ setup, I have a publisher that both sends and subscribes to events, and a separate consumer application. When the consumer is down and the publisher sends and processes an ...
- 90
1
vote
1
answer
147
views
Azure Queue Trigger function instantly moving messages to poison queue
So I have quite a basic Azure HTTP function that creates job messages whenever I do a POST request to the url:
using System;
using System.IO;
using System.Threading.Tasks;
using Microsoft.AspNetCore....
- 41
2
votes
0
answers
285
views
Cloudflare queues with slow consumer issues
I'm having problems with my Cloudflare queue consumers. Despite using the different configuration options: (max_batch_timeout=0, max_batch_size=1, delaySeconds=0), my workers take between 5 and 10 ...
- 65
0
votes
1
answer
53
views
Any HTTP/HTTPs supported Messaging or Queuing system available?
Friends,
I am looking for Messaging/Queuing service, preferably opensource that supports Queuing-Dequeuing over HTTP(its a must). I know there are a lot of Cloud based services(AWS SQS, Azure EventHub)...
- 337
0
votes
0
answers
122
views
How to define RabbitMQ queue using spring-cloud-streams without a listener
I need to set the exchange and queue for RabbitMQ in my Spring Boot application, the thing is, The application only acts as a producer, but I need to define de queue as well in the service.
steps that ...
- 306
1
vote
1
answer
75
views
Designing long running transactions
I have to conduct tasks 1-6 in an orchestrated async manner. every task has a receive queue and respond queue. These tasks are long-running...can take several days to finish.
The orchestrator is ...
- 65
0
votes
1
answer
53
views
Message queue and synchronization
I have one thread that is waiting in a message queue. Let's call it THFS.
Other threads send messages to this queue, asking for some operations.
These operations need a response. I'm not sure what is ...
- 37
0
votes
0
answers
34
views
Making Queue class singleton is incorrect?
QueueFactory
/**
* QueueFactory.ts
*
/
/**
* `QueueType` is an enum representing the different types of queues that can be created using the `QueueFactory`.
*/
export enum QueueType {
RABBITMQ =...
- 101
0
votes
0
answers
31
views
How Can I Mock Asynchronous Microservices?
I'm looking for a solution to my problem. I want to build a mock server to handle communication between some microservices. There are both synchronous and asynchronous communications between them, and ...
-1
votes
1
answer
119
views
Sping Kafka @KafkaHandler ignores message
I am developing a project with Spring Boot + Kafka. I'm having hard times with understanding why my @KafkaHandler not catching the event I'm sedning through windows commandline. To send message I'm ...
- 364
0
votes
2
answers
100
views
Job scheduling and deduplicating events in an in-memory task queue pattern with .NET 8
I have a database of messages that I want to send. Each has a unique message identifier and is placed in the database by another process. The only communication between the two processes is via the ...
- 697
1
vote
0
answers
52
views
How to Aggregate Asynchronous Messagges from a Queue into a Final Result at Scale?
I'm working on a system where:
A producer sends approximately 100 million messages daily to a message queue.
The consumer processes each message from the queue and produces multiple parts as output.
...
- 222
0
votes
0
answers
139
views
.NET database-backed queues
I'm creating an API in .NET 8 for small uses and I need to implement 3 or 4 queues for long processing that don't depend on other systems besides its own database, like RabbitMQ.
I've only found the &...
- 180
0
votes
0
answers
21
views
IPC using Message Queue random jumps in ids
I recently came across this topic while going through Advanced Programming in Unix Environment about IPC using message queues. I implemented a simple program that creates and deletes queues 5 times in ...
- 1
1
vote
0
answers
231
views
How to access ipc on host from a docker container and keep internal ipc private?
An application let's say A (which is made of several processes) uses a lot of hard coded filenames for message queues, shared memory and semaphores. The application is proprietary, should be run as ...
- 2,360
0
votes
1
answer
348
views
Azure Service Bus - Differentiating between Messages on the same Topic
I'm relatively new to Azure Service Bus (and message queueing in general), and have run into technical detail that seemed like a simple question but without a solution clearly available online.
The C# ...
- 606
0
votes
0
answers
29
views
Remove Specific MSMQ Message
Message message = mq.Peek
if (message != null)
{
message.Formatter = new XmlMessageFormatter (new string[]{ "System.String,mscorlib"});
string messageBody = message.Body.ToString ()...
- 1
1
vote
1
answer
370
views
How to implement RabbitMQ fanout type exchange in NestJS?
I'm currently working on integrating RabbitMQ into my monolithic NestJS application for real-time inventory management as part of my e-commerce app. I want to use a fanout type exchange to broadcast ...
- 119
0
votes
3
answers
194
views
How can you force a blocked mq_receive call to give up?
I am trying to upgrade a messaging library from SYSV message queues to POSIX message queues. Unlike with SYSV queues, closing and/or deleting a POSIX message queue that is blocked in mq_receive() ...
- 1,266
1
vote
1
answer
114
views
Does flush.interval=1 guarantee data durablly written to disk in Kafka?
The default durability model of Kafka is to write "in memory" to multiple brokers at which point the produce request is acknowledged. Writes (and fsync) to durable storage comes later, and ...
- 66.3k