Questions tagged [distributed-system]
This tag can be used by questions on distributed system concept, design, and implementations.
217 questions
1
vote
1
answer
212
views
Prevent unregistered users from accessing the system using JWT?
I am working on an identity and users service in a microservices system for which a passwordless, SMS-based authentication is a hard requirement, i.e.
User enters their phone number
System sends the ...
- 227
5
votes
3
answers
1k
views
How to design a distributed system with an event broker where strong ordering is required?
I was trying to model a fairly simple real world model inside a distributed system and got stuck thinking about timing and order and would appreciate some external view on it.
Assuming I have this ...
- 799
-1
votes
1
answer
152
views
How should I implement System Design patterns in general and when to use these patterns in every aspect?
I have been learning a lot of new things lately, DevOps, Cloud Computing, Monitoring, and Security. I have been facing my problems dead on, but System Design seems to be a bit complicated. I have ...
- 27
3
votes
4
answers
417
views
Does possession of a valid JWT automatically imply the user is authenticated?
I am tasked with implementing authentication and authorization in a distributed environment, so I plan to use JWT.
I get how authorization works with JWT - if the token is not expired, and the ...
- 227
6
votes
5
answers
3k
views
System Design for low latency reliable online chess game
I want to build a low-latency chess backend (something similar to live chess on chess.com or basically any other online real-time chess), but there are a couple of questions I am unable to get answers ...
- 170
1
vote
1
answer
56
views
What multicast guarantees does passive replication provide?
I know that active replication provides a total ordered multicast (atomic multicast) but what does passive replication provide ?
2
votes
2
answers
151
views
For distributed workflows - tradeoffs having stateless workers operate on central state, versus stateful workers?
I'm working on a problem right now that processes incoming data at a very high rate. Each event that flows in has an association ID, and each group of associated events will affect behaviour over time ...
- 141
4
votes
2
answers
191
views
Lock management for isolation in SAGA Pattern
System Overview:
The architecture involves two services with separate databases:
User-Order Service: Handles user actions such as placing, canceling, and returning orders.
Seller-Order Service: ...
- 149
3
votes
3
answers
558
views
Design question for exactly-once processing in a message-driven system using a unique ID
To achieve exactly-once processing where messages are consumed from a queue with at-least-once delivery, many sources (e.g. here and here and here) suggest attaching a unique ID to messages in the ...
- 139
2
votes
1
answer
255
views
How to track external media (binaries) that multiple engineers will update?
I'm not sure if this is the right forum for this--if not, please point me in the right direction. This will be a little long-winded due to the specific nature of my question, so I apologize in advance....
- 137
1
vote
3
answers
159
views
Does quorum protocols circumvent fundamental limitations posed by the CAP theorem?
I can have multiple replicas of my database to ensure high availability. Then I can have a quorum such that R+W > N to ensure consistency.
So does strategies like quorum base read/write or ...
- 1,450
0
votes
1
answer
188
views
How to rebalance data across nodes?
I am implementing a message queue where messages are distributed across nodes in a cluster. The goal is to design a system to be able to auto-scale without needing to keep a global map of each message ...
- 115
1
vote
2
answers
112
views
Subscribing to a range of topics per instance of a service [closed]
If I have a system where my distributed service is sending live scores of 1000 s of football games from some hypothetical event to millions of clients. The service subscribes to the games from a ...
- 254
0
votes
1
answer
177
views
how about using kubernetes statefulset to mapping the snowflake datacener id and worker id
I am developing a distribution id project, now using the twitter snowflake id as the fundation of the distribution id. In kubernetes cluster, to fetch the uniq and non-conflict datacenter id and ...
- 103
2
votes
1
answer
329
views
Problem with data consistency in Microservice architecture(2pc transaction) when external dependency envolved
I have microservice architecture. I implement a distributed transaction flow (2-phase commit), kafka, and 1 coordinator service. So, my problem is in each step of preparation I am calling an outside ...
- 157
-2
votes
1
answer
226
views
What is the difference between Eventual Consistency, Strong Eventual Consistency, and Causal Consistency?
I read articles about CRDTs and frequently come across the terms 'Strong Eventual Consistency,' 'Eventual Consistency,' and 'Causal Consistency.' What is the relationship between these consistency ...
- 11
-1
votes
1
answer
98
views
Designing a Distributed System for Indigenous Data Sovereignty Across Nations [closed]
I'm looking for some quick "back-of-the-napkin" thoughts from systems engineers on the following scenario:
A collection of Indigenous groups (in the US, Canada, and Australia) want to design ...
6
votes
4
answers
2k
views
Protecting against malicious duplicate IDs in a distributed environment
Let's say we have multiple (somewhat autonomous) (micro-)services, and when entities are created, the ID (UUIDs or whatever) can be set externally. How can we ensure that an ID remains unique across ...
- 71
0
votes
0
answers
83
views
Distributed caching system that scales and enables real-time UI updates?
I have a NextJS app for which I want to cache data from our backend APIs to reduce the load when we have high volumes of concurrent users scrolling through (or search) years-worth of historical data.
...
- 9
1
vote
1
answer
205
views
Architectural decision regarding at-most-once semantics cron jobs in distributed systems
please note that while I refer to a specific web application framework in the following, the problem also arises with most other web application frameworks I know so please don't be afraid to reply ...
- 119
0
votes
2
answers
241
views
Fault tolerance in aggregated distributed state
I have a scheduling system that is horizontally scaled, and stores shared state in a redis key.
The purpose of the system is to implement something similar to classic rate limiting, but a bit ...
- 21
1
vote
2
answers
336
views
Generating unique number in distributed environment
I have little app that i plan to run behind load balancer, with up to 5-8 instances.
Inside the app has service, that needs to generate unique number that has up to 10 digits ( so 1-10 digits) each ...
- 227
1
vote
2
answers
149
views
What is the starting point of a geo-distributed server farm?
Scenario:
A web application that wants to implement geographically high availability (HA), has multiple web servers in different cities around the globe. Each server is connected to a database in the ...
- 121
1
vote
1
answer
179
views
App that can be deployed both as monolith and microservices?
We have a monolith C++ application and I consider doing some major refactorings. Since we're not completely ready to go full "microservices only" (maybe in about two years time), I want to ...
- 493
2
votes
2
answers
1k
views
How to effectively scale one microservice to multiple instances
Let’s say I have a booking microservice (single instance) with its own DB.
If I want to horizontally scale the service with multiple instances(exact same copy of the business logic code) running on ...
- 157
-2
votes
1
answer
213
views
Rate limiting design
I am designing rate limiting in my application in which the rules can be dynamic. I have a basic design in which every request is validated against the rate limiting quota, which is stored in redis. ...
- 199
4
votes
4
answers
450
views
How do replicas of an app instance assure the task was done only once when the main instance fails?
Let's assume i app have with basic flow:
Client send request to server
Server do some business logic
Server puts message into some message queue
Server returns success to the client
microservice A ...
- 407
2
votes
3
answers
1k
views
How to synchronize cache writes in a distributed system?
I am looking for a complete pattern implementing cache-aside when used in a distributed system across multiple nodes read/writing from/to cache.
Specificly how to avoid making multiple requests to db ...
- 145
1
vote
3
answers
490
views
What algorithm can I use in order to not let there be two rows with same "mobile number" in database?
The issue that I'm facing is on a banking app. There are two ways to register for internet banking.
A) Self Register
B) Ask for bank to register via their GUI panel backend.
Consider this scenario:
...
- 191
-2
votes
1
answer
113
views
SERVICES in sending ENTITIES in distributed systems
Here is a passage from the Blue book:
Distributed system architectures, such as J2EE and CORBA, provide
special publishing mechanisms for SERVICES , with conventions for
their use, and they add ...
- 37
1
vote
1
answer
646
views
How does a flow of architecture of a digital banking system looks like?
I am working at a digital banking company i.e company that provides software for banks so that the customers of that bank can use banking services via smartphone.
But the documentation there is non-...
- 191
-1
votes
3
answers
638
views
Avoid race condition using RDBMS transaction
I am working on design of a distributed system which will process some events. For simplicity lets say, multiple instance of same service will be consuming from same queue, every message will have a ...
10
votes
3
answers
3k
views
How do distributed databases follow unique constraints?
Lets say i have an application where user can register, and the username has to be unqiue value.
Now lets say i have N partitions and for each partition i have M replicas with multiple leaders.
Now i ...
- 407
1
vote
1
answer
1k
views
GRPC over internet
I want to build a grpc server which will be run by a company (service provider) and multiple client each run by a different company (a client). I thought of this instead using a Rest API to get ...
- 13
3
votes
0
answers
509
views
Best practices for managing software releases with CI/CD with distributed software components running on a customer on-site machine?
I'm looking for some resources on learning how to best manage our software releases in an efficient manner, as our current method slows release schedules down.
My current team produces and administers ...
- 47
1
vote
2
answers
94
views
Phantom read race condition keeping RDBMS in sync with external storage
Here's an interesting scenario, consider a cache with many buckets, and resources that can be shared between buckets:
Bucket
Highest to lowest priority
Foo
A, B, C, D
Bar
B, C, D
Baz
A
In the example ...
- 111
0
votes
2
answers
163
views
How to handle the consumption of multiple event types when timing is crucial?
I have a distributed platform which allows customers to make purchases, and the items which are purchased are stored in an inventory:
Sales app -> PurchaseEvent -> Inventory app
The Sales app ...
- 333
2
votes
2
answers
158
views
Should a mobile app keep an 'outdated' version of the data and refresh it from database or always fetch from database?
I'm building a mobile, in which the user is able to create,modify and delete entries in a database. There are multiple screen where some of the entries are displayed: (here are some examples, not ...
0
votes
1
answer
203
views
Redux Pattern in a distributed system?
I would love an opinion on system design in a distributed system that operates under the same domain.
I have a bunch of microservices following the Database per Microservice pattern, and updated via ...
- 149
0
votes
1
answer
1k
views
Many-To-Many relationships in a Microservice architecture and if the services are correctly identified
I am creating a distributed chat system with a microservice architecture where a user can log in and chat with another user. The services that I identified is a conversation service that handles chat-...
- 11
0
votes
1
answer
171
views
Should I create access control as a microservice or keep the access control logic as part of the specific microservice?
I was wondering if I have a microservice called, for example, Weather, and only certain people can update certain properties around the weather such as Temperature, Precipitation, and Wind.
These ...
1
vote
1
answer
477
views
Determine when distributed and parallel process is completed
In a distributred system that uses parallel processing to complete tasks and I want to determine when all sub tasks have been completed, what design method or principals can i apply ?
I am using ...
- 149
0
votes
0
answers
86
views
Handling competing requests for the same "id" in different Kubernetes PODs
I have the following problem, I am implementing a delivery management service and a single order can be competed between N deliveryboys, this service is being executed on Kubernetes with 5 PODs of ...
0
votes
1
answer
325
views
Capture a time-ordered sequence of item-level modifications via events when using Amazon Aurora
I'm designing a small platform based on a series of event-based micro-services. The persistence storage I'm targeting is (the managed) Amazon PostgreSQL (Amazon RDS for PostgreSQL) — although I can go ...
user175557
0
votes
1
answer
116
views
How to design a fifo coupon distribution system
I have an app written in java and using mysql as DB. I need to create a coupon distributed system where I have 100 coupons stored in DB and the first 100 customers who login should get those coupons. ...
- 109
3
votes
2
answers
274
views
How to propagate seldomly changing data in a distributed system
We store data – some kind of metadata like car manufacturers or types of computer parts – in one of our applications.
This data changes rarely. Let's say it changes once every two weeks.
This data is ...
- 207
3
votes
0
answers
277
views
Microservice architecture but with strong data consistency [closed]
I'm working on a project that has microservice based architecture.
Recently we started facing problems where a single user requests requires updating multiple microservices each having it's own ...
- 139
0
votes
0
answers
56
views
How to handle multi-region writes for a joint user registration?
UserA and UserB are signing up for some joint service together. They both have to independently agree to T&Cs before sign-up is complete. Once complete they both receive an email to sign-in.
A 1 ...
4
votes
3
answers
519
views
How to think about system design questions methodically?
I'm working on my first big system design project at work for the first time in my career. In the past, while i've worked on large systems, I was always working with someone else, so I wasn't really ...
- 345
3
votes
1
answer
974
views
Reprocessing out of order events in a stateful stream
Say that you have a monitoring application that reports downtimes for a large number of remote systems (think IOT). A monitoring daemon polls the remote system and reports that status (on/off) via an ...
- 756