MPI_Scatter dynamic 2d array rows result in segmentation fault

Question

I have a dynamic 2d array which is allocated in contiguous memory space, however if I try to scatter this array into two MPI process using MPI_Scatter will result in segmentation fault, the entire codes are pasted here:

dynamic_2d_array.h

#ifndef _DYNAMIC_2D_ARRAY_H_
#define _DYNAMIC_2D_ARRAY_H_

#include <stdio.h>
#include <stdlib.h>

//typedef double real;
typedef float real;

real **allocate_dynamic_2d_array(int nrows, int ncols);
void free_dynamic_2d_array(real** array_dynamic);
void print_matrix(real** array_dynamic, int nrows, int ncols, char* fmt_string);

real** allocate_dynamic_2d_array(int nrows, int ncols) {
    /* here is the method to correct the non-contiguous memory problem */
    int i;
    real** array_dynamic=(real**)malloc(nrows*sizeof(real*));
    real* data=(real*)malloc(nrows*ncols*sizeof(real));
    for (i=0; i<nrows; i++){
        array_dynamic[i]=&(data[ncols*i]);
    }
    return array_dynamic;
}

void free_dynamic_2d_array(real** array_dynamic){
    free((void*)array_dynamic[0]);
    free((void*)array_dynamic);
}

void print_matrix(real** array_dynamic, int nrows, int ncols, char* fmt_string) {
//void print_matrix(real array_dynamic[][4], int nrows, int ncols, char* fmt_string) {
    int i,j;
    for (i = 0; i < nrows; i++){
        for (j = 0; j < ncols; j++){
            printf(fmt_string, array_dynamic[i][j]);
        }
        printf("\n");
    }
}

#endif // #ifndef _DYNAMIC_2D_ARRAY_H_

Here is the scatter_mat.h:

#include <stdio.h>
#include <stdlib.h>
#include "mpi.h"
#include "dynamic_2d_array.h"

int main(int argc, char ** argv)
{
    MPI_Init(&argc, &argv);
    int rank, psize, root = 0;
    int i,j;
    int ncols;
    int M=48,N=3;
    real *sub_mat;
    real **A;

    MPI_Comm_size(MPI_COMM_WORLD, &psize);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    if (rank==0) {
        A = allocate_dynamic_2d_array(M,N);        
        printf("before scatter:\n");
        int idx=0;
        for (i=0;i<M;i++){
            for (j=0;j<N;j++) {
                A[i][j]=idx++;
                printf("%4.1f\t",A[i][j]);
            }
            printf("\n");
        }
    }

    ncols = M/psize;
    sub_mat = (real*)malloc(N*ncols*sizeof(real));

    MPI_Scatter(&(A[0][0]),N*ncols,MPI_FLOAT,sub_mat,N*ncols,MPI_FLOAT,root,MPI_COMM_WORLD);

    for (i=0;i<ncols*N;i++)
        printf("%3.1f\t",sub_mat[i]);
    printf("\n");

    if (rank==0) {
        free_dynamic_2d_array(A); 
    }

    free(sub_mat);
    MPI_Finalize();
    return 0;
}

If I compile it using mpicc and run it using mpirun -np 2, it will result in segmentation fault for rank 1, what confuses me even more is that it works well using debug mode (mpicc -g) however crashes under release mode, I guess there must be some memory access issue however I cannot figure that out, could anyone give some advice?

Below is the MPI compiler information:

$ mpiexec --version
mpiexec (OpenRTE) 1.6.2

Report bugs to http://www.open-mpi.org/community/help/
$ mpicc -show
icc -I/usr/local/packages/openmpi/1.6.2/Intel-13.0.0/include -L/usr/local/packages/openmpi/1.6.2/Intel-13.0.0/lib -lmpi -ldl -lm -Wl,--export-dynamic -lrt -lnsl -libverbs -libumad -lpthread -lutil

Thanks a lot!

What I have found last night is that it seems MPI had some trouble Scatter the dynamic 2D array pointers, so current a workaround is using another float* pointer to point to A by defining a float *sendp; at the beginning and then add sendp=&(A[0][0]) after A = allocate_dynamic_2d_array(M,N);, then replace &(A[0][0]) with sendp in MPI_Scatter, this at least does not return segmentation fault. — Roy Li
– Roy Li, Commented May 14, 2016 at 17:53

gudok · Accepted Answer · 2016-05-14 07:03:14Z

2

The problem is that you are dereferencing &(A[0][0]) in all ranks but it is allocated only in root rank. Pass NULL as first argument to MPI_Scatter in all ranks except in root one.

You should also rename ncols to nrows (semantically).

answered May 14, 2016 at 7:03

gudok

4,2092 gold badges23 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Roy Li Over a year ago

Thanks for your answer however it does not seem to be the cause, actually I have already tried adding a A=NULL before if (rank==0), this still results in segmentation fault. In MPI_Scatter, all A pointers other than the root is ignored, is that right?

Roy Li Over a year ago

In addition, Scatter is a collective operation, we cannot do something like: if (rank==0) MPI_Scatter(xxx), how do you make the judgement if?

gudok Over a year ago

Does this work? MPI_Scatter(rank == root ? &(A[0][0]) : NULL, ...)

Roy Li Over a year ago

Yes, it worked, however is this the recommended way to do this? If I pass a float *sendp=&(A[0][0]) to MPI_Scatter it will work as well...

gudok Over a year ago

The problem is not in MPI_Scatter but in evaluating first argument of it which dereferences A. A doesn't point to valid memory in non-root ranks and that causes segfault. If dereferencing A works then this means that either A points to valid memory (by luck) or compiler optimizes code in such way that dereferencing doesn't occur at all.

Collectives™ on Stack Overflow

MPI_Scatter dynamic 2d array rows result in segmentation fault

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related