Questions tagged [scalability]
Scalability is the ability of a system, network, or process, to handle growing amount of work in a capable manner or its ability to be enlarged to accommodate that growth.
216 questions
0
votes
2
answers
92
views
How can I efficiently handle click events and sequentially related operations (like user–URL deactivation) in a scalable URL shortener?
I am learning backend development by building a simple URL shortener project. My goal is to learn backend concepts and try to do things in simple but better ways, keeping scalability and reliability ...
1
vote
3
answers
472
views
Practicing DDD from the biggest modules to the smallest widgets
I have a frontend project using VueJS and TS and I structured it using DDD. So far, I have some modules (bounded contexts, domains - use the term you prefer) which are used in two ways: some modules ...
0
votes
1
answer
95
views
Best practices for serving encrypted files in Laravel?
I'm building a Laravel-based web application that functions as a personal online photo archive. Each user uploads and manages their own private collection of images. These images are encrypted at rest ...
- 11
0
votes
2
answers
176
views
Scaling/microservices approach to reading files from same directory
My company receives files via SFTP. We currently have a service running on a timer that:
polls the inbound directory
moves files to an 'In Progress' directory
processes files (queueing messages for ...
- 3
0
votes
1
answer
131
views
How to create a job management for a hybrid cloud/on-premise software?
I am exploring the state-of-the-art methods to create a service that can run and scale in both cloud (container) and on-premise environments.
The current version of the software is designed for on-...
- 139
-2
votes
3
answers
322
views
How do I build my application with high throughput in mind? (Need guidance)
I am building a Spring boot REST API app that is part of a microservice architecture project. What I am planning:
My app listens for events from two other services and after some business logic, ...
- 1
1
vote
1
answer
239
views
Scaling Heavy Writes When Fetching Data From External API
I have a service that fetches financial data from an external APIs and parses and saves that data in a postgres DB. The data can be quite large in size, our containers have crashed quite a lot due to ...
- 21
1
vote
1
answer
143
views
How do you build a high availability, horizontally scaling application [closed]
I've designed an application, but I've had problems with scaling. Initially, this application was a simple docker compose file, that ran on a decently powerful VM.
Containerization is great, but the ...
- 119
-1
votes
1
answer
492
views
How to make a hierarchical permission architecture work performantly?
I was going through this blog - https://www.notion.so/blog/data-model-behind-notion
Which talks about how each page can inherit permission from parent page.
we use an “upward pointer” — the parent ...
- 115
1
vote
2
answers
246
views
Designing a microservice architecture
I am looking to enhance my skills in back-end technologies and would need your help in setting up a scalable microservices architecture.
Here's my project.
I have N sensors that send me data (Pressure,...
- 11
1
vote
1
answer
206
views
Design considerations for data shared across multiple users: is Redis a good candidate?
I have an app that has an assets table like the following:
user_id
code
current_price
1
ALUP11
12.5
2
ALUP11
11.9
user_id and code are unique together and current_price is updated if the user triggers ...
- 121
1
vote
4
answers
1k
views
Is it okay for API endpoints services to communicate directly with the database?
I've started work on a web application personal project that is more a learning opportunity more than anything and I'm brainstorming the stack. My number one priority is to not overcomplicate things ...
- 161
3
votes
1
answer
579
views
Scaling up an online game using SignalR
I'm developing a web game that currently all runs on one server. It uses SignalR to maintain connections between the server and clients, and the server also sends all the web stuff to clients (HTML/...
- 1,318
1
vote
0
answers
484
views
Do we really need a message brocker in WebSocket system?
I'm engineering architecture of a new web based software.
I've never worked on high-scale softwares before and I'm reading a lot about it.
To increase client-side speed and reduce load on servers, and ...
- 19
35
votes
10
answers
10k
views
Am I right that switching programming languages will have little impact on the scalability of a CRUD API?
An external consultant to our team advised us to rewrite our SaaS offering (essentially a CRUD API) in .NET because this is more "scalable" than using Node.js (or Ruby on Rails, Flask, etc.)....
- 544
0
votes
1
answer
369
views
Architecture solution for scalable crawling
I have the following requirements:
a list of queries (rather long list ~ 100K - 1M items or even more) that I need to perform in a certain period of time and the execution time must be predictable. ...
1
vote
0
answers
111
views
Handling notifications at scale
How do applications like twitter handle notifications at scale.
For example @BarackObama has 131.3 million followers. When he tweets, how is that notification sent to all 131.3 million followers.
I ...
- 119
3
votes
0
answers
102
views
Difficulty understanding how federation increases cache performance for databases?
I am studying system design for distributed systems and in this page (https://github.com/donnemartin/system-design-primer), one of the following advantages was mentioned for federation for databases ...
- 153
5
votes
2
answers
3k
views
Real time chat at scale
I have been looking at different solutions for large scale chatting solutions. I feel as if I understand 90 % of it but am turning to this forum to tie the knot.
I imagine running a bunch of message ...
- 189
-3
votes
1
answer
77
views
Server performance and CMS scalability [closed]
I have been given the job of a colleague who has resigned and I don't know where to start. I hope you can give me some hints:
It is about a CMS made from scratch (Javascript/jQuery/PHP/MySQL) which ...
- 97
-1
votes
1
answer
59
views
Scalable Web Data Service Approach
I'm looking to build a service that interacts with an existing web application's database. The service would retrieve data from an api for which the web application users own an account and warehouse/...
- 1
0
votes
1
answer
450
views
Low Latency for multi-million record database in a web app [closed]
As a forewarning, this is for a school project with other school teammates, so my question is just for the sake of learning best practices. We are creating a recommendation system web app which ...
- 103
0
votes
1
answer
288
views
How can users be notified of changes during collaborative editing in a system like Google Docs?
"Designing Google Docs" is a popular system design interview question, and there are plenty of articles on it on the web. However, those mostly touch on merging changes, but don't explain ...
- 132
0
votes
1
answer
129
views
designing high throughout system for storing hundred thousands incoming records per second in SQL server database
In our company we have a requirement where we would like to store hundred thousands incoming records per seconds. we currently a pub-sub model for processing many records(100/sec) from many system(~...
- 107
0
votes
1
answer
215
views
How to deal with massively concurrent events?
I got the following interview question:
Say the company has this event, where we will create a button on a website and only the first ten clicks will be accepted and win a prize. Assume that there a ...
0
votes
2
answers
131
views
Start small - but design in such way that sharding is possible - how?
Following question is more about best-practices than a real problem - nevertheless, I'd like to know how to do it in best way.
Given a service, that can operate in multiple countries/geo-areas, one ...
3
votes
1
answer
1k
views
Is consistent hashing required for sharding?
I am reading about scaling of database and came to know about sharding technique. But I also read about consistent hashing technique. So how practically sharding is implemented? Do we arrange nodes in ...
- 321
2
votes
0
answers
64
views
Scaling Order Management System for ecommerce
Tech Stack: I am using MySQL 8 with InnoDB engine. Application is built using Java with Spring & Hibernate. Database is hosted in AWS RDS and web apps in EC2.
I have two primary tables for ...
- 259
1
vote
1
answer
241
views
How can an anycast IP address handle millions of RPS (requests per second)
So, in terms of systems design:
given a highly scalable system being served through a single load-balanced anycast IP address,
when that system receives a load in the order of (hundreds of) millions ...
- 119
6
votes
3
answers
342
views
How can adding redunancy adversely affect performance
I have been reading about system design and was going through this popular article - A Word on Scalability
The author here says:
An always-on service is said to be scalable if adding resources to
...
- 171
2
votes
1
answer
148
views
Sharding rule updating on DB scaling
When you have sharding, horizontal partitioning, you have multiple DBMS instances, and your server app chooses which one to use by some rule that uses record IDs. Well, for example, we have ...
- 191
6
votes
4
answers
1k
views
Performance implications of a game project monorepo on *modern* Git
First off please note, I’m fully aware that Git has historically been terrible at dealing with large binary files and very large repos; there are at least a dozen questions on SO that deal with large ...
- 251
0
votes
0
answers
56
views
Scalable database design review
I'm doing some research on how to scale our current SQL database and need some advice with a possible solution. The goal being able to handle more data and doing it in a way that performs well.
We ...
- 218
2
votes
2
answers
3k
views
How do atomic updates work at scale?
I am working on a weird ID system where I generate all the IDs in advance and need to atomically pop() one from the database, so no ID is read twice at the same time (and so used more than once). This ...
- 2,787
-3
votes
2
answers
72
views
How should I handle a scalability problem during routing my requests from server?
I have faced a problem with the initial design of my system where sensitive information was being sent to the front-end and front-end was responsible for calling 3rd party APIs. As you have probably ...
- 103
-1
votes
3
answers
205
views
Can I manage thousands of concurrent connections with a non-Node stack?
I'm developing a social network with Django (Python) + Postgres SQL. My site will have a chat feature so that users can communicate to each other in real-time, and the communication will be only from ...
- 9
-4
votes
1
answer
167
views
application can not handle certain amount of RPS when 3k+ sessions connected
i have a problem in my scalable Pub/Sub application that it can not take more than 30 RPS when big amount of sessions are open against it.
first i will explain the application structure.
the ...
- 21
0
votes
1
answer
110
views
How to make sure all of the nodes process one API request behind a load balancer
I have an architectural question. I have an application which is subscribed to a log compacted Kafka topic. I have to process each event and store into a persistent datastore. I am planning to run the ...
- 101
-1
votes
2
answers
129
views
Developing your application as microservices or as a monolithic app - which is cheaper when deploying and scaling?
I am developing an application, and want to implement a certain functionality. I find that this can be done in the stack I am using, however it's hard to implement. I can create a microservice with ...
- 123
2
votes
2
answers
2k
views
Designing a scalable microservice: how to prevent one instance from performing an action if another instance allready did?
As I am moving a part of the monolith app logic to a microservice, I am standing before a problem with scalability.
Currently, the main monolith runs on different instances, and has some scheduled ...
- 129
3
votes
2
answers
691
views
Architecture for real time updates from the data in file system
I am trying to revamp my legacy application to make it scalable and performant. Its current architecture is something like this
Consider a short-lived script that gets invocated 500k+ times a day, ...
- 119
3
votes
3
answers
1k
views
How can I scale a database that requires that writes are consistent?
Suppose I have a table that saves plane ticket purchases. I need to keep track that I don't oversell so when a user purchases a ticket it needs to be stored in that table and other customers need to ...
- 141
4
votes
1
answer
263
views
How does a push-based(write fanout) feed system handle unsubscribe logic, message erasure and new following user's message
I am curious about these three questions when building a push-based(write fanout) feed system:
How does it handle the unsubscribe logic?
To me, it seems to me that we have two choices: delete or ...
- 305
-3
votes
2
answers
103
views
How to scale an application where every read results in a write?
So I am trying to design a simple URL shortener application where every time a URL is queried, it is going to update the number of times it has been queried.
I'm thinking of using MongoDB and I am ...
- 291
1
vote
1
answer
218
views
Is my server design safe regarding multiple threads and concurrent database reads/writes?
I'm making a chat server using sockets and a MySQL database, and after it's working I want to expand it to become a more complex game server.
I want to know whether my design is missing anything.
...
- 111
5
votes
4
answers
3k
views
Should we share a database between two 'Microservices'?
We're implementing a service that exposes data related to a particular part of a business. It will pull data in from different sources, do some ETL, and store it in Redis. It will expose this data ...
- 159
0
votes
1
answer
411
views
Is event-sourcing and event-driven design works in scalable architecture for a wallet?
considering you need to design a simple wallet service that keep tracks of transactions, and give balance for each account.
I started designing the solution using event-driven architecture, so every ...
- 11
10
votes
4
answers
17k
views
Should data transformation be on the front or on the back end in this scenario?
Let us say I have some data in a Json format. Let us say the Json format is the following:
{title,
resources[] }
The resources array contains information about a graph. But it should be iterated upon ...
- 1,159
1
vote
0
answers
41
views
How would i design processing twitter data
I am working on a project where a user can input different criteria which will be used to fetch tweets, lets call this action as TweetAnalysis. These tweets will then be sent to another internal ...
- 245
0
votes
0
answers
359
views
What could be a good scalable alternative to polling data and then looping over a list of objects for triggering some action on those?
So I have some naive implementation for the following problem:
We have a list of objects with action methods that have to be trigger at some value of interest.
This value is polled (or is streamed) ...
- 101