4

I wrote a hybrid openMP/MPI program and I call it like the following

mpirun -np ncores --bind-to none -x OMP_NUM_THREADS=nthreads ./program

where ncores is the number of non shared memory processes (MPI) and nthreads is the number of shared memory threads (OpenMP).

That means in each of the ncores, the program will be executed on nthreads.

I do not want to have nthreads in every core but I am rather interested in varying that number for each core. e.g. if ncores=2 I´d like to set 2 threads on core 1 and 6 threads on core 2.

Is there a way to do that?

I am using Open MPI 1.10.3

1
  • 1
    What is the configuration of your computing system: number of nodes, number of cpu cores on each node. You just can vary both arguments in mpirun line: ncores and nthreads. If some of your MPI process (with id 0) want to have 2 threads and other MPI process (id 1) wants 6 threads, don't use global environment variable, switch to omp_set_num_threads function Commented Feb 17, 2017 at 22:38

1 Answer 1

4

You has global environment setting "OMP_NUM_THREADS" for all your MPI processes. This setting is the same for every of them, and you want to have different thread number for various mpi processes, so you should not set it as global variable.

I can suggest two variants. First is to delete -x option and replace your "./program" with some bash script "./program_script.sh", which will get process id from mpirun (use mpirun -np 2 env to find name of variable with process id) and set correct value of "OMP_NUM_THREADS" env variable for mpi process with this id and then start your ./program.

Second variant is to delete -x option and set threads number directly from your MPI processes using standard omp_set_num_threads() call of OpenMP Standard:

MPI_Comm_size(MPI_COMM_WORLD,&nproc);
MPI_Comm_rank(MPI_COMM_WORLD,&rank);
if(rank == 0) {
   omp_set_num_threads(2);
}
if(rank == 1) {
   omp_set_num_threads(6);
}

Don't forget to set correct CPU binding in your mpirun (check with --report-bindings, set with --bind-to or rankfile).

And third variant (when your thread count is less or equal to cpu core count only) is to delete -x option and just set correct cpu binding from your mpirun: enable core 0 and 1 for 1st MPI process and cores 2,3,4,5,6,8 for your 2nd MPI process. Most OpenMP libraries will sense allowed cpu set and start by default one thread per CPU.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.