21,952 questions
0
votes
2
answers
44
views
How to run requests paralelly in JMeter transaction controller?
I have login transactions in my JMeter which has 20 requests in it. few requests run parallely and few of them in line when I observed in developer tools network tab.
My issue, jmeter is running all ...
2
votes
1
answer
160
views
How can I know if this piece of code is thread safe or not?
I've written a piece of code that aims to encode into and decode base64 characters. So far, my code is working, but I've been made aware of the risk of a data race in a previous code of mine, and I ...
0
votes
1
answer
63
views
Rayon Threadpool, why is the exit thread call back not executed?
I am trying to learn about rayon's thread pool. And to that effect I want to make a simple program where a set of tasks is added to the pool, and then when the last task in that batch executes it adds ...
0
votes
0
answers
19
views
Asynchronous/Parallel end-point and Flask
REST API is based on HTTP, which is inherently synchronous, meaning, a client makes request, and he is meant to get the full answer (from OSI application layer point of view), and not some request ...
1
vote
0
answers
80
views
loop parallelization with omp
Good morning, I was trying to parallelize a loop with OMP in a program I created for a flood simulator with the goal of achieving a performance boost. The loop I want to parallelize is as follows:
#...
0
votes
0
answers
45
views
Parallelism for many dataframes with sparksqlcompare
I have many number of data frames up to 10. I need to compare 10 data frames with other 10 data frames which are independent to each other. Each of data frame has a specific keys to be compared. ...
-2
votes
1
answer
159
views
How to Enable Parallel Threads for Copying non-numeric Data from SQL Server to Azure Storage
I'm attempting to copy a table from an on-premises SQL Server to Azure Storage using a data pipeline in Fabric. The table consists of 5 columns and 1000 rows, with all columns being of type varchar. ...
1
vote
0
answers
65
views
Live logging with doParallel and cmdstanr
I have defined a Bayesian A/B test power analysis (sample size calculator) using cmdstanr.
Since it is a computationally expensive process, I tried to save output as each loop ended in foreach.
...
3
votes
0
answers
98
views
CUDA Dynamic Parallelism Synchronization (CUDA > 11.6)
I need to use a function like cudaDeviceSynchronize to wait for a kernel to finish execution. However after version 11.6, it is no longer possible to use any form of synchronization within device ...
0
votes
0
answers
82
views
In the nim-lang, I want to pass a ref to a read-only tree structure to the spawn'ed nim-taskpool
Sorry for the long message, but I was wondering if you could give me some advice.
What I want to achieve is to handle tree-structured data in the nim language using tasks with the nim-taskpool package....
0
votes
0
answers
104
views
How to automatically adapt database connection in Rust integration tests without modifying function signatures?
I am encountering an issue with writing integration tests in Rust for an application that uses a database (PostgreSQL managed via SeaORM).
Some of my tests need to use the database to be properly ...
2
votes
1
answer
150
views
Error: a __device__ function call cannot be configured
I want to implement CDP for a basic forward function (I will call the forward function too many times at the same time (also from a CUDA function) and because of that I want to use CDP)
Here's the ...
0
votes
0
answers
74
views
Paralellizing a Genetic Algorithm in Python is slower
I've been coding a Genetic Algorithm to solve TSP using python's DEAP library. I've implemented my own Graph class, and an Algorithm class that runs the GA on a Graph instance. However, when adding ...
0
votes
0
answers
36
views
Can one receive simultaneous progress updates from a `doFuture` multisession using `progressr`
I have a slow running function that I execute in multiple settings in parallel using doFuture together with foreach. I would like to receive regular progress updates from each execution using ...
-2
votes
1
answer
133
views
Is there any parallel computing involved in scipy.linalg.solve? [closed]
I tried to understand how scipy.linalg.solve works and if it involves parallel computing. If yes, can someone give me more information?
Also, I would like to know: if we want to compute the inverse ...
1
vote
0
answers
83
views
How to efficiently modify blocks in Rust array using rayon in parallel?
I'm trying to build a scientific computing software with Rust, which requires manipulation of the matrix during operation. A typical matrix operation is to append sub-blocks of small matrices to a ...
0
votes
0
answers
253
views
Unable to achieve multithreading in WSL2
I'm trying to execute the following C++ multi-threaded program on WSL2:
#include <iostream>
#include <pthread.h>
#define N 1024*1024*1024
#define num_threads 8
using namespace std;
long x[...
1
vote
1
answer
100
views
Why is wait in bash is not being respected? [closed]
I have written a bash script that:
gets the latest Github CI/CD run
checks the digest of downloaded artefacts
spawns a process to download and extract each artefact (in parallel) if digest has ...
2
votes
0
answers
78
views
Poor performance when incrementing Fortran coarrays
Consider three versions of the following Fortran code using coarrays
Version 1:
program test
implicit none
integer :: dp = kind(0.d0)
complex(dp), allocatable :: G(:,:,:,:,:,:,:)[:] ...
0
votes
0
answers
55
views
Optimizing Bulk Image Download in Flutter (5000 Images in Under 30 Seconds)
I am working on a Flutter project where I need to download 5000 images (each 2500x2500 px) as fast as possible, ideally within 30 seconds. These images are fetched from Lorem Picsum and stored in the ...
-1
votes
2
answers
146
views
Bash asynchronous processing with COPROC
I'm trying to get to run two parallel asynchronous processes in a loop, one main and one via COPROC, in order to be able to collect and synchronize results, when both have finished (for each loop), to ...
2
votes
0
answers
48
views
Extending a Single-Pass Scan Kernel for Independent Row-wise Scan in CUDA
I implemented a kernel that performs a single-pass scan (proposed in the book Programming Massively Parallel Processors):
__global__
void SinglePassKoggeStoneScan(const unsigned int * input, unsigned ...
0
votes
0
answers
68
views
Slurm parallelization using python?
There is a job I want to run on slurm that needs parallelization because it is otherwise too slow. Even though the computations are independent of each other, they don't happen at the beginning of the ...
0
votes
1
answer
35
views
Error in R: "one node produced an error: Argumento NA/NaN" when extracting raster values in package parallel version 4.2.2
I'm working on an R script to build a correlation matrix based on values extracted from WorldClim bioclimatic layers at specific geographic coordinates (WGS84).
I have a dataset (db) with 842 obs. of ...
0
votes
1
answer
166
views
MSTest parallel execution: Tests run in alphabetical order in Visual Studio but not in Azure DevOps pipeline
I'm using MSTest with parallel execution at the class level in my .runsettings file. Some test classes handle web tests, while others perform the same operations without web tests.
The Issue
In ...
0
votes
0
answers
52
views
Deadlock issue when scheduling task computations (F#, .NET TPL)
I am in the process of building an effect system DSL with F# inspired by ZIO and Cats Effect.
I have my effect type defined as a DU of opcodes, a fiber type, a channel type for message passing, etc. I ...
0
votes
0
answers
82
views
How to Efficiently Process a Concurrent Queue Using a Thread Pool in Java?
I'm working on a Java application where I need to periodically check a MongoDB collection (CacheBilgi) and enqueue new data into a concurrent queue (cacheInfoQueue). Then, I need to process this queue ...
0
votes
0
answers
46
views
How do I wake up the worker nodes when driver node is doing all the work?
I am running a very simple script in Databricks:
try:
spark.sql("""
DELETE FROM raw.{} WHERE databasename = '{}'""".format(raw_json, dbsourcename))
...
1
vote
0
answers
26
views
PowerShell - How to make defined constants and functions accessible in ForEach-Object -Parallel
I'm working on a PowerShell script that needs to use ForEach-Object -Parallel to run several iterations of a process simultaneously. Here's my code (with names changed to protect the innocent). Right ...
0
votes
1
answer
55
views
The thread running under multiprocessing.Process does not update its instance attributes [closed]
I want to run several class instances in parallel and update the instance attributes using Redis. On each class instance, I run a thread in the background to listen to redis for changes. When I run ...
0
votes
0
answers
30
views
Xarray apply_ufunc with input of several 3D arrays
I have a function apply_dask_function() that I use to apply the function dask_function() on a xarray dataset (ds_example).
dask_function() takes 3 inputs, which are two 1D arrays (length "time&...
0
votes
0
answers
40
views
Copula garch seems to be stucked in infinite loop
I am estimating a static Copula-EGARCH model using the rmgarch package in R. My dataset consists of 58 univariate time series over 2274 days. The model specification is as follows:
# Static Copula-...
3
votes
1
answer
70
views
How to run on.exit if user aborts/interrupts the calculation?
Question: In R or RStudio, if a user aborts an ongoing calculation (by clicking the stop sign or pressing esc), on.exit is not called. Is there a way such that if the user presses esc, a certain ...
1
vote
0
answers
40
views
Slurm: how to wait for scripts without affecting parallelism
I have a problem running my slurm code (I am quite a newbie).
I am trying to run slurm with tasks in parallel in the same job, lets call them task1, task2, ..., task28, with jobs in parallel.
The srun ...
0
votes
0
answers
164
views
Running pyscf parallelly using mpi4py
I am trying to use PySCF for electronic structure calculations and was curious about the parallelization schemes available to get the best performance.
I've installed PySCF via Spack, which included ...
2
votes
1
answer
135
views
Why does my file have race conditions, even though I used StandardOpenOption.SYNC?
I have a CSV File that is too large to fit in RAM, and only barely fits on my hard disk. I need to split this CSV file into parts.
I am using BufferedReader.lines() to stream the file, and now I would ...
0
votes
1
answer
124
views
Mental Model for Hybrid MPI/OpenMP with SLURM
Question
I am trying to develop a clear mental model for using SLURM to request resources on HPC systems for hybrid MPI/OpenMP jobs. In thinking about it more, I realized there are some gaps in my ...
0
votes
0
answers
66
views
How to set up parallel computing with progress bar using the doParabar package in R
I'm having some difficulty setting up parallel computing with a progress bar in R. I tried using the doSNOW package following the solution here, but no progress bar appeared. The code also took over ...
0
votes
1
answer
97
views
Parallelize a list of subsequent api calls in Python
I have some code that cycles through some files and then makes subsequent calls to a "chain" of multiple APIs using contents from each file, where each API needs the output of the previous ...
1
vote
1
answer
55
views
R parallel inside of function
I regularly make use of the packages parallel and pbapply. However, I have come across some odd behavior that I assume is by design, but I can't figure out how to work around it. Basically, if I use a ...
1
vote
1
answer
108
views
Performance issue using HPX for parallelization in C++ code
I am trying to parallelize my code using HPX in order to improve performance. Below is the original code and my attempt to refactor it using HPX.
Original Code:
std::vector<std::vector<std::pair&...
0
votes
0
answers
50
views
MPI Collective communication along axes with uneven data distribution per rank
I am attempting to implement a method in MPI for a well established particle simulation program that involves image processing. The program runs a loop for millions of iterations that performs a ...
0
votes
0
answers
47
views
parallel foreach in R is not faster with more cores on an AWS EC2 server
I have to run a simulation study for a model we developed in my phd. I have 64 simulation settings consisting of combinations of different parameters and sample sizes (1000 or 10,000), and I want to ...
0
votes
0
answers
61
views
Future_sapply from future.apply R package does not work on Linux
I am trying to use parallelism for my code, when I execute it on Windows everything runs fine, but I need to use a Linux cluster and there all the values are just NaN, I executed every individual part ...
0
votes
1
answer
102
views
Should memfrac in terra be adjusted for the number of workers in parallel?
I have some code in terra that I'm running using the future/future.apply package but I'm running into some memory issues (Error in eval(expr, p) : std::bad_alloc). I'm using the future package's "...
-1
votes
1
answer
95
views
How to avoid repeated CSV reading in each parallel process in Python?
I am working on a Python project where I need to process a large CSV file using Polars. Currently, I am reading the CSV file inside each parallel process, which works but is inefficient because the ...
0
votes
1
answer
69
views
How can I improve this implementation of a foreach-parallel function in PowerShell v5?
I found this interesting function, ForEach-Parallel that works in PowerShell v5. It's part of the repo chocolatey-oneget-provider at GitHub that I've tried to improve upon.
The reason I'm not using ...
0
votes
1
answer
81
views
Assessing the contribution of communication to the runtime of an MPI program
Background
Let's say I have a complex MPI program with multiple message passing events and computations. The communication pattern is that of bidirectional ring messaging as shown in the figure below.
...
0
votes
0
answers
85
views
Some CSV outputs are empty in parallel Python processing, even though the code path should be triggered
I have a Python script that processes a batch of CSV files in parallel. Each input CSV has exactly 200 rows, so I use an if num_videos == 200: check to run a function like:
if num_videos == 200:
...
0
votes
0
answers
21
views
Calling multiple times a function that uses Rmpi
I would like to repeatedly use a function that calls on clusters from Rmpi, but I am having issues where the first function call hangs indefinitely.
Here is a simple code example:
library(parallel)
...