1

I have an MPI application that currently has one process (call it A) which is causing serious problems for scalability. Currently, all the other processes are sitting in an MPI_Recv waiting for that one process to send them information.

Since I want to speed this up now with as little effort as possible, I was thinking about using OpenMP parallelize process A. Is this practical?

Since the other processes sharing a node with A are in an MPI_Recv, can I utilize all the resources from that node to work on process A, or will the MPI_Recv prevent that?

The other benefit of using OpenMP is that the memory can be shared since process A takes a lot.

By the way, does it change anything if my processors are waiting in an MPI_Send instead of an MPI_Recv?

Improve this question
2
  • Do you want something like this OpenMPI and OpenMP Example? Commented May 29, 2012 at 3:04
  • @baccus -- Something like that, but, say (for example) that I have a machine with 10 nodes, 4 cores per node -- On 9 nodes, I want to run 4 mpi processes, on the 10th, I want to run only one MPI process (which spawns a few more via openmp) Commented May 29, 2012 at 9:57

2 Answers 2

Reset to default
3

Yes, it is possible to use OpenMP to parallelize a certain process locally combined with OpenMPI that takes care of work distribution (i.e. OpenMPI across nodes and OpenMP within nodes). This concept is known as Hybrid Programming with OpenMP and MPI (if you google for this you will find several useful links).

MPI_Send and MPI_Recv calls are blocking calls (for detailed information you can check this post In message passing (MPI) mpi_send and recv “what waits”), which means that if your nodes are blocked in MPI_Recv they will be blocked waiting for data. However, you can use the respective asynchronous methods MPI_Isend and MPI_Irecv for performance at the cost of having to deal with race conditions and careful buffer handling. An example and further information can be found here.

In my opinion you have two choices:

  1. Evenly distribute your workload using OpenMPI and then use OpenMP to parallelize your workload locally (if you have several cores and several nodes with several cores you can use OpenMP to assign tasks to each core; OpenMPI to distribute parts of it through the nodes which can then take advantage of the local architecture of each node and use OpenMP);
  2. Reprogram your program to use the asynchronous methods in order to have other nodes helping node A in its computations if necessary.

I hope this helps.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

Yes, I understand the difference between blocking and non-blocking communication. I guess I was wondering if sitting in a blocking communication call will (typically) consume all the CPU resources for that core. I suppose if I don't want to consume all the resources I can use non-blocking followed by a MPI_Test/sleep combo...but that is quickly becoming more involved than I want it to be. (I was really just looking for a quick hack to get things done now while I continue to work to make it scale better)
I was wondering if you could do the following: we know that MPI_Recv() blocks until the size of the message, source and tag matches. So a simple test before calling MPI_Recv(), such as checking the size of the message, would say if we should call it or if we can help other nodes while waiting for the message. For instance, each second the message size is checked and if it doesn't match you are able to allocate resources to help other nodes. It's just an idea.
actually, the size doesn't have to match and you're describing a perfect use-case for MPI_Irecv()+MPI_Test(). But that approach probably won't work here anyway. The "fast" nodes don't have the information they need to help the "slow" node. My ultimate goal is to take 1 slow node and turn it into a number of fast nodes. That works, but the memory doesn't scale well (the slow node needs a HUGE amount of memory) -- so I thought a shared memory paradim might help. Ultimately, there are some things I can do to help the memory issue -- it's just a lot of work. Thanks though.
I got the idea after reading this detailed explanation of MPI_Recv(), but after comparing to the ones detailed in the OpenMPI website I agree with you.
@baccus : But the idea is still sound -- You can MPI_Iprobe for messages from a particular source on a particular tag and then you can also get the size of the message if you want it from there...Ultimately I just don't think that model will work in my application. :-/
|
0

Using OpenMP and MPI is relatively easy, if I understand you correctly is should speed things up. But the whole thing looks like a hack. Have you considered to redesign(fosters design methodology) your program? Having one node block all the others is not a good design.

Improve this answer

2 Comments

Yes, having one node block the others is not good -- that's what I'm trying to prevent. The best solution is to parallelize the work that the one (slow) node is doing so that it works as fast as the others ... There are a couple problems with that though. That one node takes alot more memory than the others, so parallelizing it (each process needs to use all that memory) which doesn't scale well using MPI. It could probably be optimized, but not without a decent amount of effort -- OpenMP seems easier to churn out results while working on the better solution.
Check the threads compatibility of the MPI library. MPI defines four levels of threading support and which of them are implemented by Open MPI is selected at configure/compile time. You'd be safe if you do not communicate from inside OpenMP parallel regions.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.