0

I'm trying to handle some dynamically-allocated multidimensional arrays in C++ using MPI. To avoid worrying about non-contiguous memory, I've written a class wrapper which allows me to access a 1d array as if it were 2d. I'm trying to create an MPI data type to send instances of the class through MPI_Send.

My code is below. When I send each element of the class in its own MPI_Send buffer, it gives the expected results. When I try to use my custom MPI data type, it gives a segmentation fault. By commenting/uncommenting a few lines, you can try both ways.

The class uses arrays right now, but I also get the same results with vectors. By commenting/uncommenting a few lines, you can try that as well.

#include "mpi.h"
#include <stdio.h>
#include <vector>
using namespace std;

//. arrays will be this big
const int N(2);

//. this class is a lightweight wrapper around a 1d vector so that it can be accessed as 2d. this
//. ensures that the memory use is contiguous, so it can be sent through mpi. by commenting/
//. uncommenting, it can be set to use either a vector or an array. 
template <class type> class arr2d{
  public:
    int s[2]; //. size (length and width)
/*    vector<type> v; */ //. vector data container 
    type* v; //. array data container
/*    void init(const int& s0, const int& s1){s[0] = s0; s[1] = s1; v.resize(s[0]*s[1]);} */
    void init(const int& s0, const int& s1){s[0] = s0; s[1] = s1; v = new type[s[0]*s[1]];}
    type& operator()(const int& i, const int& k){return v[s[1]*i + k];}
};

int main(){
  //. standard mpi stuff
  int mpi_rank, mpi_size;
  MPI_Status stat;

  //. declare an arr2d object
  arr2d<double> x;
  x.init(N,N);

  //. displacements, types, and elements (for mpi_type_create_struct)
  MPI_Aint     disp[2];
  MPI_Datatype type[2];
  int          elts[2];

  //. this will hold the arr2d mpi data type
  MPI_Datatype mpi_arr2d;

  //. fire up mpi
  MPI_Init(NULL,NULL);
  MPI_Comm_rank(MPI_COMM_WORLD,&mpi_rank);
  MPI_Comm_size(MPI_COMM_WORLD,&mpi_size);

  //. put some values in the rank 0 version of x
  if(mpi_rank == 0){
    for(int i=0;i<N;i++){
      for(int k=0;k<N;k++){
        x(i,k) = i+k+0.5;
      }
    }
  } else { //. rank 1 starts with x full of zeros
    for(int i=0;i<N;i++){
      for(int k=0;k<N;k++){
        x(i,k) = 0;
      }
    }
  }

  //. displaceemnt of elements of x (vector implementation)
/*  disp[0] = (int*)&x.s - (int*)&x;
  disp[1] = (int*)&x.v.front() - (int*)&x; */

  //. displaceemnt of elements of x (array implementation)
  disp[0] = (int*)&x.s - (int*)&x;
  disp[1] = (int*)&x.v[0] - (int*)&x; 

  //. types of elements of x
  type[0] = MPI_INT;
  type[1] = MPI_DOUBLE;

  //. quantities of elements of x
  elts[0] = 2;
  elts[1] = N*N;

  //. assemble and commit mpi_arr2d
  MPI_Type_create_struct(2,elts,disp,type,&mpi_arr2d);
  MPI_Type_commit(&mpi_arr2d);

  //. check what each rank sees before communication
  printf("rank %d sees %f %f %f %f \n",mpi_rank,x(0,0),x(0,1),x(1,0),x(1,1));

  if(mpi_rank == 0){
    MPI_Send(&x,1,mpi_arr2d,1,123,MPI_COMM_WORLD); 
/*    MPI_Send(&x.s,2,MPI_INT,1,124,MPI_COMM_WORLD); */ //. send just the size
/*    MPI_Send(&x.v.front(),N*N,MPI_DOUBLE,1,125,MPI_COMM_WORLD); */ //. send just the vector
/*    MPI_Send(&x.v[0],N*N,MPI_DOUBLE,1,125,MPI_COMM_WORLD); */ //. send just the array
    printf("just send to rank 1\n");
  }
  if(mpi_rank == 1){
    MPI_Recv(&x,1,mpi_arr2d,0,123,MPI_COMM_WORLD,&stat); 
/*    MPI_Recv(&x.s,2,MPI_INT,0,124,MPI_COMM_WORLD,&stat); */ //. recv the size
/*    MPI_Recv(&x.v.front(),N*N,MPI_DOUBLE,0,125,MPI_COMM_WORLD,&stat); */ //. recv the vector
/*    MPI_Recv(&x.v[0],N*N,MPI_DOUBLE,0,125,MPI_COMM_WORLD,&stat); */ //. recv the array
    printf("just recved from rank 0\n");
  }

  //. check what each rank sees after communication
  printf("rank %d sees %f %f %f %f \n",mpi_rank,x(0,0),x(0,1),x(1,0),x(1,1));

  MPI_Finalize();
  return 0;
}
2
  • 1
    By commenting/uncommenting a few lines, you can try both ways So is the version you posted, without any further tinkering, the one that works or the one that doesn't work? Commented Mar 1, 2015 at 17:02
  • The version above does not work. Changing int* to char* makes everything work (per Dr. Tower). Commented Mar 3, 2015 at 19:13

1 Answer 1

1

I think the issue is that you have (int*) pointer casts when calculating your displacements, when the displacement values need to be in bytes. I was able to get it to work using (char*) pointer casts when calculating the disp[0] and disp[1] values.

(void*) wouldn't compile for me.

Sign up to request clarification or add additional context in comments.

1 Comment

That worked for both the array and vector implementations. Thanks!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.