21,952 questions
1
vote
2
answers
71
views
How to debug OpenMP scalabilty
I'm trying to implement Conway's Game of Life in C, showing that it can scale with OpenMP. The results in the following picture are from running it on a socket of a machine with 64 cores equally ...
- 13
0
votes
0
answers
31
views
AzureML - SAS Token Expiry for Parallel Processing Job
We are running an AzureML job with a parallel component. The component is instructed to look into a URI_FOLDER path and distribute the operation contained inside the run() method of the entry script ...
- 148
2
votes
1
answer
132
views
How to log each iteration in a parallel function
I am migrating LTO-6 tapes to LTO-9 tapes. I wrote a script for this that uses parallel so that I can have two or 4 tape drives running at the same time. It works totally fine but the logging. I was ...
- 391
0
votes
0
answers
45
views
How to use set_config() method of sktime objects
I want to perform parallel computing on my ShapeletTransform. According to the documentation, I need to set the backend:parallel flag to 'loky' to activate the parallel method of the class. However, ...
0
votes
1
answer
85
views
How can I proceed with parallel imputation after encountering non-convergence in `futuremice`?
I am conducting a multiple imputation in a relatively big dataset (over 700 variables, over 10.000 records), using mice, specifically futuremice function.
impDt <- futuremice(data,m=5,maxit=20,...
- 916
0
votes
2
answers
108
views
Problems with looping - Start-Threadjob use in Powershell [closed]
POSH Code: https://pastebin.com/sKYCJSpZ
Line 819: Remove-Job -Job $j
This is a very long script that cycles through forests and domains and pulls lists of users and groups (with their membership) ...
- 13
0
votes
0
answers
111
views
Implementing lock-free remove for hashmap
I'm trying to implement a lock-free hashmap in C.(It doesn't actually free the elements when removing, I'm letting memory leak until it's destroyed). A node contains value, next ptr, and char array ...
- 1
1
vote
1
answer
61
views
Correct pattern to process ZeroMQ messages using Goroutines
I am implementing a Zero MQ message listener service in Go, and I am struggling to find the most idiomatic pattern of processing those messages and storing them in a DB using Goroutines. I am ...
- 1,503
0
votes
1
answer
76
views
mpi4py: only rank 0 participating in scipy minimize after first iteration
Given the code below, I am unable to remedy the fact that only rank 0 participates in evaluations of Objective after the first iteration of the (SciPy) minimizer. Obviously the rank!=0 workers finish ...
- 1,195
0
votes
1
answer
43
views
Parallel sliding window computations with fixed intervals?
Sharing a common DolphinDB use case and solution for data processing.
How to implement a script in DolphinDB to perform sliding window calculations on a vector at equal intervals?
For example, ...
- 319
3
votes
0
answers
141
views
C++ CUDA: Adding a kernel printf changes results of next kernel printf [duplicate]
ENVIRONMENT: Windows 10; VS 2022 C++; Compute Capability 6.1; Cuda 12.8.
I saw CUDA was being used at work on a project, so on my own time, I am trying to learn CUDA programming by watching NVIDIA ...
- 325
3
votes
1
answer
107
views
Does omp parallel for scheduling affect memory access?
I have a found myself most confused by the result of my tests regarding effects of different scheduling for #pragma omp parallel for. Basically my confusion sprouted from the following problem:
I have ...
- 41
-4
votes
1
answer
287
views
Is Bryce Lelbach's claim regarding progress guarantees on non-NVIDIA GPUs true?
In a talk on The C++ Execution Model, from the cppunderthesea 2024 conference, at around 44:50, NVIDIA's Bryce Adelstein Lelbach claims, that non-NVIDIA GPUs give no guarantee of threads progressing (&...
- 137k
-1
votes
2
answers
85
views
What's the most efficient way to calculate word counts for 4m records and a word list containing 1 and 2 grams?
Here is my problem. I have a very large file with about 4m rows, where each row has something like a paragraph of text. Then I have a word list with about 150 terms, some are 1-grams and some are 2-...
- 89
1
vote
1
answer
124
views
Efficiently processing millions of files in a .NET 8 Console app on Debian Docker (Linux mounted Windows Share)
I'm working on a .NET 8 console-based application running in a Debian-based Docker container, which scans a Linux-mounted Windows Share file system. The application needs to process 2–3 million files ...
0
votes
0
answers
25
views
azure synapse Processing time
When my Azure pipelines are triggered automatically, my processing time takes 1 hour... when I run it manually, it takes 30 minutes... why this difference could be happening... even if my pipelines do ...
- 1
0
votes
0
answers
109
views
How can I properly load a LoRA weight into a pretrained Stable Diffusion model on TorchServe and enable parallel inference?
I'm attempting to serve a pretrained Stable Diffusion model with LoRA weights applied using TorchServe. However, the LoRA weights don't seem to load properly, and I'm not sure why. Could anyone help ...
- 1
2
votes
1
answer
66
views
Can I use sa-learn from Spamassassin in parallel with mutliple CPUs?
I am writing a bash script for using Spamassassin.
I have a single folder with loads of emails stored as text files. (1744408703.1184229_99.txt). I want to feed all of them to Spamassassin through sa-...
- 387
1
vote
1
answer
144
views
Improving performance of a threadpool
I am tying to write my own threadpool, I am noticing that currently, trying to use this custom made threadpool is slower than doing the exact same work in a single threaded fashion and I don't ...
- 9,991
1
vote
1
answer
62
views
Does running parallel and async together cause errors in java?
**Config TaskExecutor:**
@Bean("threadImport")
public TaskExecutor threadImport() {
log.info("*** init threadImport with corePoolSize [{}] maxPoolSize [{}]",
...
- 124
0
votes
0
answers
62
views
Why does my Python multiprocessing code run slower than the single-threaded version?
I'm working on a CPU-bound task in Python and attempted to speed it up using the multiprocessing module. I expected performance improvements due to parallel execution, but instead, the multiprocessing ...
0
votes
1
answer
86
views
MPI Parallelization using the master-slave model
I am currently writing a c++ MPI program to parallelize a certain computation. On Rank # 0, I have a priority queue of jobs. What I want to do is as follows. First, the highest priority item in the ...
- 169
0
votes
0
answers
50
views
XGBoost GPU version not outperforming CPU on small dataset despite parameter tuning – suggestions needed
I'm currently working on a parallel and distributed computing project where I'm comparing the performance of XGBoost running on CPU vs GPU. The goal is to demonstrate how GPU acceleration can improve ...
- 19
2
votes
1
answer
86
views
Writing from particles to grid in parallel without locks efficiently [closed]
I have the following setting. I have a set of particle sin a simulation and a grid with cells.
Each particle needs to update a neighbourhood around itself in the grid.
Thus I can think of two options:
...
- 9,991
0
votes
0
answers
71
views
How to access a package loaded with devtools::load_all() when using foreach?
I'm working on an R package 'mypkg' and use devtools::load_all() inside an Rstudio project to simulate what loading it would look like after making changes. This works fine for all my code except for ...
- 763
1
vote
2
answers
95
views
What is the difference between an MPI nonblocking collective write, iwrite_all vs a "nonblocking" noncollective iwrite combined with a file sync?
I'm setting up IO for a largescale CFD code using the MPI library and the file IO is starting to eat into computation time as my problems scale.
As far as I can find the "done" thing in the ...
- 15
0
votes
1
answer
199
views
How can I optimize my 'foreach' loop to read and process each line of a text file more efficiently in C#?
I have C# code that reads a .TXT file, processes each line, and subsequently inserts documents into a MongoDB database. Since the file contains many lines (approximately 70k), both reading all lines ...
- 19
0
votes
0
answers
71
views
Extracting WAIC using rjags::jags.samples() from single chain of bayesian model run in parallel with JagsUI
I am using JagsUI to run many single and multi-predictor models, using the autojags() function to run them until convergence. I would like to do model selection using WAIC, rather than DIC. The JagsUI ...
- 15
0
votes
0
answers
50
views
Filter collection asynchronously/in parallel?
Consider this:
enum State { OK, Maybe, No };
State CheckState(string s) { /*Some logic*/ }
void Work() {
HashSet<string> myStrings = GetStrings();
var result = myStrings.Where(s => ...
- 1,623
0
votes
1
answer
103
views
How to check the parallel degree of tables and indexes in OceanBase?
In OceanBase (MySQL Mode), I know that I can set the parallel degree of a table or index using SQL commands like:
alter table my_table parallel 4;
alter index my_index parallel 2;
However, for ...
user30132773
0
votes
1
answer
74
views
How can I scale the number of concurrent workers based on the available CPU (Windows)?
My current implementation tries to figure out the available CPU in percents and then spawn new workers as long as there is at least 5% available.
Here is the code:
public class CpuAvailabilityMonitor
{...
- 63.6k
0
votes
0
answers
70
views
Does the mirai package use shared memory parallelism?
Does the mirai R package use shared memory parallelism?
The parallel package uses shared memory parallelism with the mc* (e.g. mclapply) functions and fork clusters, in the sense that the input data ...
- 1,150
2
votes
1
answer
67
views
Solving for cost-optimal processes for bucket sort using Ts/pTp = 1
I am doing a class on parallelization. We're given an MPI program and told to determine the maximum big O for processes so that the program is cost-optimal.
Cost optimality is defined where big theta ...
- 23
-3
votes
1
answer
82
views
Crystal Report generates reports correctly in parallel on IIS but throws System.Runtime.InteropServices.COMException in NUnit test
When running NUnit test to check if multiple PDF reports have been generated correctly the following exception is being thrown System.Runtime.InteropServices.COMException : Invalid index. (Exception ...
- 138
2
votes
1
answer
145
views
Powershell script error when using parallel processing
I'm using PowerShell with -Parallel to speed up a script that checks file formats using ImageMagick. The script works without parallel processing, but it's too slow with a large number of files. When ...
- 45
2
votes
1
answer
79
views
Error in parallel processing in R when using the atakrig package
I am using the atakrig package. When setting the number of cores to 8 for parallel computation I am getting this error: 0%Error in serialize(data, node$con) : error writing to connection. If I set the ...
- 413
0
votes
0
answers
103
views
Pathos multiprocessing map not completing on Linux machine, works on Mac M2
I am using the pathos multiprocessing to parallelize a gradient calculation that is embarrassingly parallel, using finite difference.
Below is a high level example of how it is set up,
`
...
class ...
- 1
2
votes
4
answers
323
views
Is it possible to create a task without starting it, so I can limit max concurrency?
I have a weird scenario. I have a list of groups. For each group, I need to start a task for all the contents, but I need to make sure the last value in each group is run last. Simplified function ...
- 188
1
vote
0
answers
53
views
Setting the stack size for nested parallel regions with Open MP (Fortran)
I'm using Open MP with Fortran 08 (compiled with GFortran) to create nested parallel regions (in my case, a 3-level nest). Before running my executable, I set some Linux environment variables to ...
- 11
1
vote
1
answer
113
views
Why do OpenMP programs run faster on a single process than on multiple processes?
The task was to implement various matrix multiplication algorithms using OpenMP. It turned out that with num_threads(1), the program runs faster than with any other number of threads. Is this due to ...
0
votes
0
answers
54
views
How to run unit test parallel in fastlane iOS
Is there any useful plugin to run parallel unit test in fastlane for ios?
lets say like i have 10 schemes, currently its running in loop but it is taking too much of time. so i wanted to run parallel ...
0
votes
1
answer
95
views
Raster calculations parallelisation on hpc using a shiny app
I am currently developing a shinyapp for rastercalculations similarly like:
https://github.com/NEPEM-UFSC/plimanshiny
I am developing it to run on a high performance cluster because I want to ...
- 1
0
votes
1
answer
75
views
Parallel processing with arguments to modify
I need to parallelize a function that modifies values of one of the arguments. For example modify this simple code:
def Test(i, a, length):
if i % 2 == 0:
for x in range(0, length, 2):
...
- 5,979
0
votes
0
answers
46
views
SSAS Tabular 1600 - Partition Processing Issue - Random Empty partitions durring Full parallel processing
SSAS Tabular 1600 - Partition Processing Issue - Random Empty Partitions During Full Parallel Processing
Details:
We have a fact table with 800 million rows, split into 25 partitions by Month Index (...
- 1
1
vote
0
answers
114
views
Loss of precision in Parallel.For loop [duplicate]
I am trying to perform calculations on a large (300000+ items) array of doubles. I have tried iterating through this array with a regular for-loop. This works, but is rather slow. I then tried using a ...
user30064726
2
votes
1
answer
132
views
How to declare an OpenMP reduction for a std::vector inside a struct?
I'm trying to perform an std::vector sum reduction with an OpenMP reduction declaration for it:
// g++ -fopenmp MRE.cpp -o MRE
#include <vector>
#include <algorithm>
#include <omp.h>
...
1
vote
0
answers
71
views
How to make flask asynchronous to handle multiple users at the same time?
I spun up a cloud server and put my flask web app code plus a frontend in there. When a user runs an operation on the frontend this gets sent to the backend endpoint as a JSON, then processed and ...
- 11
3
votes
1
answer
84
views
Measuring the on-screen displayed width of strings in a parallel set-up
In order to adjust the columns' widths of a spreadheet document to their content, I need to compute the 'real' width in 'characters' (defined by Microsoft as the ratio of the real width divided by the ...
- 494
2
votes
0
answers
43
views
NVIDIA webinar on parallel reduction gridsize
In the last example of Mark Harris' webinar I don't understand the indexing before the parallel reduction part. In "Reduction #6" the gridSize/number of dispatches was ceil[N (the size of ...
- 71
0
votes
3
answers
339
views
Convert multiple-page PDF files to PNG quickly
I have a folder containing 600 PDF files, and each PDF has 20 pages. I need to convert each page into a high-quality PNG as quickly as possible.
I wrote the following script for this task:
import os
...
- 801