Initializing MPI cluster with snowfall R

Question

I've been trying to run Rmpi and snowfall on my university's clusters but for some reason no matter how many compute nodes I get allocated, my snowfall initialization keeps running on only one node.

Here's how I'm initializing it:

sfInit(parallel=TRUE, cpus=10, type="MPI")

Any ideas? I'll provide clarification as needed.

openMPI, no error message as far as I know (things run smoothly, but the sysadmin tells me it's running it all on one cluster even though I have 5 allocated). Also I just realized I meant one node not one cluster. — user1480248
– user1480248, Commented Jul 27, 2013 at 16:11
any ideas on what could be causing it? Do I have to use socketHosts perhaps? — user1480248
– user1480248, Commented Jul 27, 2013 at 20:19

Steve Weston · Accepted Answer · 2013-07-28 15:55:20Z

7

To run an Rmpi-based program on a cluster, you need to request multiple nodes using your batch queueing system, and then execute your R script from the job script via a utility such as mpirun/mpiexec. Ideally, the mpirun utility has been built to automatically detect what nodes have been allocated by the batch queueing system, otherwise you will need to use an mpirun argument such as --hostfile to tell it what nodes to use.

In your case, it sounds like you requested multiple nodes, so the problem is probably with the way that the R script is executed. Some people don't realize that they need to use mpirun/mpiexec, and the result is that your script runs on a single node. If you are using mpirun, it may be that your installation of Open MPI wasn't built with support for your batch queueing system. In that case, you would have to create an appropriate hostfile from information supplied by your batch queueing system which is usually supplied via an environment variable and/or a file.

Here is a typical mpirun command that I use to execute my parallel R scripts from the job script:

mpirun -np 1 R --slave -f par.R

Since we build Open MPI with support for Torque, I don't use the --hostfile option: mpirun figures out what nodes to use from the PBS_NODEFILE environment variable automatically. The use of -np 1 may seem strange, but is needed if your program is going to spawn workers, which is typically done when using the snow package. I've never used snowfall, but after looking over the source code, it appears to me that sfInit always calls makeMPIcluster with a "count" argument which will cause snow to spawn workers, so I think that -np 1 is required for MPI clusters with snowfall. Otherwise, mpirun will start your R script on multiple nodes, and each one will spawn 10 workers on their own node which is not what you want. The trick is to set the sfInit "cpus" argument to a value that is consistent with the number of nodes allocated to your job by the batch queueing system. You may find the Rmpi mpi.universe.size function useful for that.

If you think that all of this is done correctly, the problem may be with the way that the MPI cluster object is being created in your R script, but I suspect that it has to do with the use (or lack of use) of mpirun.

edited Jul 28, 2013 at 15:55

answered Jul 28, 2013 at 3:42

Steve Weston

19.7k4 gold badges62 silver badges78 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

user1480248 Over a year ago

Thanks, I think the problem is most likely that I did not use mpirun (was not aware of this). So the documentation for using mpirun is as follows from the instructions for the cluster: # Invoke mpirun. # Note: $NSLOTS is set by OGS to N, the number of processors # requested by the "-pe mpi_M_tasks_per_node N" option. # If M=4, possible N are: 4, 8, 12, 16, . . . # (see Table 2 in Runningjobs page) mpirun -np $NSLOTS mpi_program arg1 arg2 .. If I want to have snowfall do all the work for me I should just do -np 1?

Steve Weston Over a year ago

@user1480248 As mentioned in my updated answer, I think you do need to use '-np 1' with snowfall. Rmpi-based packages are rather unusual in using MPI spawning, so the usual advice on how to set -np is wrong.

user1480248 Over a year ago

So if I'm allocated 16 compute nodes and I want to utilize all 16 cores on each of these nodes (which the system gives me access to) if I let cpus <- mpi.universe.size() it'll run it on all those compute nodes and also fully utilize them?

Steve Weston Over a year ago

@user1480248 If you've requested and been allocated 16 nodes each with 16 cores, then I expect that mpi.universe.size will return a value of 256. I also expect that setting "cpus" to 256 will result in 16 workers being spawned on each of those 16 nodes. But that's a guess since I'm not very familiar with OGS and it depends on having OGS support built into your installation of Open MPI.

Collectives™ on Stack Overflow

Initializing MPI cluster with snowfall R

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related