2,306 questions
1
vote
2
answers
71
views
How to debug OpenMP scalabilty
I'm trying to implement Conway's Game of Life in C, showing that it can scale with OpenMP. The results in the following picture are from running it on a socket of a machine with 64 cores equally ...
- 13
0
votes
1
answer
72
views
Websocket load test utilising high cpu and memory of the VM used for testing
I'm running api load tests with a script that's a combination of REST and websocket APIs. The resource utilisation of the VM where I'm using jmeter to execute load tests is much higher than before ...
1
vote
0
answers
55
views
Why CuFFT throughput increases as the transform size gets larger?
(Updated) I am trying to understand how CUDA parallelism works in CuFFT while learning CUDA coding.
I wrote my version of 1-D FFT in CUDA C++ and compared it with cuFFT. Below are the throughputs I ...
- 11
3
votes
1
answer
73
views
How to represent vertical scaling in UML?
It is found how to represent horizontal scaling in UML with a Layered component but nothing has been said for vertical scaling
- 83
0
votes
0
answers
36
views
How to use unique incremental integer for user ID?
|-|i
I'm making a mobile chat app
I heard about load balancing and scalability, to make it able to be used properly by multiple people at different locations of the world at same time
I'm currently ...
user22157294
1
vote
1
answer
162
views
Kafka and hotspots in a partition
I am new to Kafka and I understand that there is only guarantee of message order within one partition and not across partitions.
What I am not sure is if this can create scalability issues e.g. in ...
- 311
0
votes
0
answers
117
views
How to Handle Zoom API Rate Limit for Meeting Creation in a Multi-User SaaS Application?
I'm building a SaaS application where multiple users can create Zoom meetings. Currently, I'm using Zoom Server-to-Server OAuth to create meetings via the below endpoint
POST /users/{userId}/meetings
...
- 35
1
vote
1
answer
450
views
How Can I Implement Per-App Scaling for High-Density Hosting in Azure App Service?
I’m working on hosting multiple applications using Azure App Service, and I want to optimize resource usage by enabling Per-App Scaling. My goal is to host a high-density number of apps on the same ...
-1
votes
1
answer
73
views
Theoretically, in low-conflict scenarios , can MySQL support tens of thousands of concurrent connections without a performance collapse? [closed]
In the process of optimizing MySQL, we discovered a performance collapse at 10,000 concurrent connections during benchmarksql testing. Is there a theoretical limitation, or is it an issue with MySQL's ...
- 59
1
vote
0
answers
33
views
Optimize fetching of 923K polygons' data and computing GeoJSON
I have a complex issue with regards to scalability.
I am confused between putting load on the database, the backend server, or the frontend server.
I have 923K hexagons of levels 1-8 for France, which ...
- 106
0
votes
1
answer
37
views
no of instances of a microservice crossing the no of partitions in a topic
I am trying to learn kafka with microservices (spring boot) - now, I came across a video lecture which says that the no. of partitions in a topic is static and can't be changed dynamically - that ...
- 1,105
1
vote
0
answers
52
views
How to Aggregate Asynchronous Messagges from a Queue into a Final Result at Scale?
I'm working on a system where:
A producer sends approximately 100 million messages daily to a message queue.
The consumer processes each message from the queue and produces multiple parts as output.
...
- 222
0
votes
1
answer
349
views
Replication vs Sharding (for read scalability vs write scalability)
Can someone explain why they say db replication is more ideal for read scalability while sharding is more ideal for write scalability?
From my current understanding:
replication allows read traffic ...
2
votes
3
answers
212
views
Is using a Pandas Dataframe as a read-only table scalable in a Flask App?
I'm developing a small website in Flask that relies on data from a CSV file to output data to a table on the frontend using JQuery.
The user would select an ID from a drop-down on the front-end, then ...
- 6,893
0
votes
0
answers
52
views
Deduplication , Grouping for events table at scale
I'm working with an events table where different source tables trigger writes into this table with columns: entity_id and payload. These events are then published to a Kafka topic using a message ...
- 518
0
votes
1
answer
138
views
Efficient data processing with AWS Lambda and DynamoDB: Handling timeouts and scalability issues
I am building a serverless application using AWS Lambda and DynamoDB. The application processes a large amount of incoming data, performs some transformations, and then stores the results back into ...
- 35
1
vote
1
answer
164
views
Does WSO2 APK(API Platform for Kubernetes) have Graphical User Interface such as DevPortal and Publisher and Admin , ....?
We explore a way that has the last version of WSO2 APIM with Publisher and Devportal and Carbon and Admin , ... and at the same time have Scalable Gateway to execute for example 2000 requests per ...
- 301
0
votes
2
answers
247
views
Efficient Data Storage and Retrieval Strategy in React for High-Performance Enterprise Applications
I'm currently exploring strategies to optimize data storage and retrieval for an enterprise-level application that demands high scalability and performance. In my quest to enhance efficiency, I've ...
- 185
2
votes
1
answer
170
views
Optimizing Node.js Real-Time Notification Service for High Traffic Websites
I'm developing a real-time notification service using Node.js with Express, MongoDB for data storage, and Socket.IO for real-time communication. The service is intended to be integrated into various ...
9
votes
1
answer
10k
views
How to add millions of documents to ChromaDB efficently
I have 2 million articles that are being chunked into roughly 12 million documents using langchain. I want to run a search over these documents so I would like to have them into ideally one chroma ...
- 101
0
votes
1
answer
165
views
Horizontal scalability in Cloud Spanner
Cloud Spanner claims that its infinitely scalable database solution. How Cloud Spanner actually implements horizontal scalability under the hood?
- 57
0
votes
1
answer
87
views
Looking for alternative to nested loops in Python
I have developed the following code to check if groups of three people are conected at the same time
import pandas as pd
from itertools import combinations
data = {
'User': ['Esther','Jonh', 'Ann'...
- 503
1
vote
2
answers
52
views
Iterating over the cartesian product of a list of lists
The problem is a generalization of the folling simple problem:
Given 3 lists of elements, I can loop over the pairings between all 3 lists by folling pseudocode:
foreach( element1 in list1 )
...
- 547
0
votes
1
answer
50
views
How to store an infinite noise map into a database?
I'm thinking about designing a small 2D game where all players share the same map, generated with Perlin noise (for example), and which could be modified by players. Technically, it should be like a ...
- 47
0
votes
1
answer
738
views
How does key rebalancing in state store work in Kafka Streams with scaling out of partitions?
Let's suppose a Kafka streams that operates stateful has an input topic with 16 partitions and the number of instances (or tasks) is also 16. I understand that the default partitioner of the Kafka ...
- 51
3
votes
1
answer
278
views
Implementing scalable microservices oriented web application using Spring MVC and Springboot in Java
I am migrating enterprise web application using spring MVC and spring boot which currently working in core .Net technology. In Java I am using microservices and postgresql in backend and using angular ...
- 1,688
0
votes
0
answers
66
views
Training NN to map quantities into a grid
I created a simple NN (multi-layer percetpron acutally) in pytorch that takes a vector of quantities as an input and outputs an assignment map so that the products with the biggest quantities are ...
- 1
0
votes
2
answers
125
views
How can I scale a server with limited database connections
I've made a TCP server for a turn-based game in C#. The server uses MySql for the database and uses stored procedures for all database interactions. All database interactions are contained within a ...
- 925
2
votes
1
answer
794
views
How does scaling a web app with Docker works?
I'm learning about docker and can't seem to wrap my head around the idea of scaling with docker. Assuming 2 based docker containers, 1 for nginx, 1 for webapp (php/nodejs/etc) on 1 very powerful ...
- 23
1
vote
0
answers
381
views
JAVA Spring boot application performance issue when autoscaling
I have a java spring boot application with some open source schedulers in it. The main workflow of the application is: whenever we get the requeusts we create a task at some time as configured in the ...
4
votes
0
answers
495
views
Applying node-pressure eviction for CPU resource
I am managing a GKE cluster that runs identical machine-learning models whose CPU usage fluctuates heavily based on the current load. Thus, I made the gap between the lower and upper limit of CPU ...
- 41
0
votes
1
answer
1k
views
RAY workers being killed because of OOM pressure
I am using modin in combination with ray to read a huge csv file (56GB with 1,5 billion rows). I sorted the data beforehand using linux sort.
The following code results in multiple workers being ...
- 105
0
votes
1
answer
73
views
Is it possible to scale large data set in a local file and not overload the HEAP?
I have a vue project where I've made an aggregate data grid similar to Excel. I have a dataset, here's just one object in the array of objects as a small sample
export default [ < --- ...
- 61
1
vote
0
answers
36
views
How do I use Numpy views when doing scalable statistics bootstrapping
I have a large dataset that I process using xarray+dask for scalability. These libraries work great for all of my calculations, except for one. The final step is to perform some statistics ...
- 147
3
votes
1
answer
312
views
How can I serve ML models quickly and with a low latency
Assume a user connects via a Websocket connection to a server, which serves a personalized typescript function based on a personalized JSON file
So when a user connects,
the personalized JSON file is ...
- 879
0
votes
0
answers
58
views
Scalability : Convert large list of objects into pojo and make api call
Let's say I have a json data of multiple vendors in a large number of categories. I need to make post api calls for top 10 vendors of these hundreds of categories.
How can I do that in a scalable ...
- 77
0
votes
0
answers
84
views
Disabling iOS scaling
I have been suffering for a very long time to disable zooming with fingers, double clicks, etc. on iOS, so that it is cross-browser on all devices. Completely disable scaling on mobile phones.
I tried ...
0
votes
1
answer
138
views
How to generate and store a unique sequence in a mongo DB collection in a scaled application running with multiple instances
I've a node application deployed on Azure App Service and we've scaled it to 5 instances.
Now in our application we have exposed few API's which adds data into our mongo DB collection called detail.
...
- 1
0
votes
0
answers
79
views
Scalable way to get data ready / into pandas or consorts
I have around 600GB of csv files, around 1 billion lines, stored in around 80 million text files.
For performing additional analysis, specifically network analysis, I would have to first aggregate ...
- 105
0
votes
1
answer
442
views
What backend should I use for my react-native mobile application and Laravel portal which have to use the same database? [closed]
I am building a mobile application in react-native. I am using Firebase for authentication and I am using Firebase cloud functions with Firestore for some API functions that get information from the ...
- 619
0
votes
2
answers
671
views
How can I scale up my django app to utilize 4 cores and 8 threads?
I have a Django app. I am about to move this from a single core system to a multi core system. (8 threads because of hyperthreading)
I am trying to figure out the changes I need to make to utilize all ...
- 103
1
vote
3
answers
145
views
Optimizing a Pandas DataFrame Transformation to Link two Columns
Given the following df:
SequenceNumber | ID | CountNumber | Side | featureA | featureB
0 0 | 0 | 3 | Sell | 4 | 2
1 0 | 1 | 1 ...
- 242
1
vote
1
answer
3k
views
How to scale concurrent step function executions and avoid any maxConcurrent exceptions?
Problem: I have a Lambda which produces an array of objects which can have the length of a few thousands (worst case). Each object in this array should be processed by a stepfunction.
I am trying to ...
- 2,657
1
vote
1
answer
180
views
Distributed system - How to assign a node ID to each node in distributed system?
How can we assign a unique ID to each node/instance of an application in a distributed environment?
We have deployed our app on multiple hosts and requests are distributed across them through a load ...
- 175
-1
votes
1
answer
164
views
What will happen if I don't scale my application or database servers and I increase the load?
I understand horizontal scaling, vertical scaling, sharding, I want to gain more understanding on what will happen to the application i.e the effects of not scaling over how I can solve the problem by ...
0
votes
1
answer
185
views
How can i build a web service which loads users daily listening history from spotify and presents interesting analytics?
I have this project idea which involves using the spotify API to load users daily listening history and presenting a weekly wrapped to the user by mail, I need some advice on the architecture. My plan ...
- 15
2
votes
1
answer
564
views
Problem with instance of class while creating reusable controller in node.js
I want to reuse all my servise and controllers and I was achieving this with extending class.
I'm calling the class of Controller and service all the way up from a router.
everything looks fine while ...
- 29
-1
votes
1
answer
99
views
Google Drive Url System Design
I noticed this behavior on Google Drive.
When a link is created for a file on Google Drive, the link is valid until the file is deleted.
Moving the file to another folder(s) does not affect the ...
1
vote
1
answer
2k
views
Websocket app into stateless microservice?
i would like to convert an app i have created into a proper microservice, following all the usual practices.
However i am unable to understand how to achieve statelessness and horizontal scaling. This ...
user6342701
0
votes
1
answer
45
views
More optimized and scalable way to write code
I have following data frame as input:
FriendName,Restaurant,InterestRank,Cuisine,Cost,Alcohol
Amy,R2,1,French,1,No
Amy,R6,7,Japanese,0,No
Ben,R5,2,Japanese,1,No
Ben,R2,3,French,1,No
Cathy,R5,1,...
- 271