openmp parallel sections within omp parallel for?

Question

I have an outer for loop that I have parallelized using OpenMP. However within this for loop there are sections of code that can also be executed in parallel.

Can I use OpenMP's sections clause to parallelize this? Is this even possible? Since each iteration of the for loop is run by just one thread, can I (within each iteration), ask for certain sections of code to be run by multiple threads in parallel? Rest of the code should just be run by one thread i.e the thread to which that loop iteration has been assigned.

For ex. I have the following piece of code:

omp_p = omp_get_max_threads();
omp_set_nested(1);
#pragma omp parallel for num_threads(omp_p/2)
for(int p=0;p<omp_p/2;p++){
   size_t a = (p*N)/(omp_p/2);
   size_t b = ((p+1)*N)/(omp_p/2);
   for(int i=a;i<b;i++){
      /*Work on A[a]->A[b]*/
      for(int j=0;j<n;j++){
         for(int k=0;k<N;k++){
           /*Serial code*/
          #pragma omp parallel sections
              {
                 #pragma omp section
                   {

                   }
                 #pragma omp section
                   {

                   }

              }
           /*Serial work*/
           #pragma omp parallel sections
              {
              #pragma omp section
                   {

                   }
                 #pragma omp section
                   {

                   }
              }
           /*Serial code*/
         }
      }
   }
}

This causes the program to go much much slower than if I hadn't used the parallel sections at all..

Besides the huge overhead from nested parallelism, your i, j and k loop counters get the default sharing class of shared and should be explicitly declared private. — Hristo Iliev
– Hristo Iliev, Commented Mar 16, 2013 at 9:24
Oh I'm sorry, I forgot to declare them as ints within the for() braces. Corrected this.. — user1715122
– user1715122, Commented Mar 16, 2013 at 17:24
What prevents you from simply decomposing the loop between all threads and execute everything in the inner loops in serial? Is N too low in comparison to the number of threads? — Hristo Iliev
– Hristo Iliev, Commented Mar 16, 2013 at 18:03
That's how I had coded it but realized that apart from a few assignments (which need to be done serially), the innermost 'k' loop primarily has two big sections which can be executed in parallel. It is similar to a matrix multiplication code, with the matrices divided among threads and the chunk of the work happening in the innermost loop. So I was wondering if instaed I should give a bigger chunk of the matrix to each thread and reserve two threads (each) to perform the operations in the innermost loop in parallel. N ~ 5*omp_p — user1715122
– user1715122, Commented Mar 16, 2013 at 19:50
OpenMP, like most other parallel programming paradigms, introduces synchronisation overhead, and as such it favours computational models where each thread does as much works as possible and synchronises as seldom as possible. You implementation underutilises the threads as there are serial parts outside the nested parallelism where the second thread is doing nothing. Remove the nested parallelism, make the outer loop use all threads and then profile the code for things like false sharing. — Hristo Iliev
– Hristo Iliev, Commented Mar 17, 2013 at 0:21

veda · Accepted Answer · 2013-03-15 21:52:19Z

1

Nested OMP should be possible. But I fear that you might not see any performance gain by doing this due to the following reasons:

Nested OMP might result in generation of more number of threads than the number of CPU cores. This might end up in doing lots of context switching.
Your OMP parallel sections are deep inside 4 nested for loops, so, there might be a possibility of overhead due to creation and destruction of threads.

answered Mar 15, 2013 at 21:52

veda

6,61416 gold badges61 silver badges80 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Hristo Iliev Over a year ago

Given that the OP explicitly instructs the outer parallel region to run with half the configured number of threads and that each sections construct has only two sections, your first point is invalid.

veda Over a year ago

@HristoIliev Given the number of sections and not setting the num_threads(), Will the number of omp threads created be equal to number of sections? If yes, sorry then I didn't know about it..

Hristo Iliev Over a year ago

Sorry, obviously I have misread the modality in your text. It is implementation dependent how many threads will be used for a combined region with two sections.

user1715122 Over a year ago

Setting num_threads(2) in the parallel clause doesn't help. So I guess the slowdown is mainly due to the nested parallelism?

Collectives™ on Stack Overflow

openmp parallel sections within omp parallel for?

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related