MPI_OP_Create function

I think it's easiest to illustrate with some code. As Gilles explained, MPI will take care of all the communications and do the reduction across processes - all you need to specify is the pairwise comparison function. Note that the prototype for the reduction operation is fixed by MPI and allows for a vector reduction: the third argument is the count for the length of the vector on each process (not the number of processes which is implicitly the size of the communicator). Other than some minor issues with void vs double pointers your comparison function can be registered as-is and used for a reduction operation:

#include <stdio.h>
#include <mpi.h>

void findminnorm(void *invec, void *inoutvec, int *len, MPI_Datatype *datatype)
{
  int i;

  double *invecdble    = (double *) invec;
  double *inoutvecdble = (double *) inoutvec;

  for (i=0; i < *len; i++)
    {
      if (invecdble[i] > 0)
      {
        if (inoutvecdble[i] > invecdble[i] || inoutvecdble[i] < 0)
          {
            inoutvecdble[i] = invecdble[i];
          }
       }
    }
}


#define N 2

int main()
{
  int i;
  double input[N], output[N];

  int rank, size;
  MPI_Op MPI_MINNORM;

  MPI_Init(NULL,NULL);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Comm_size(MPI_COMM_WORLD, &size);

  MPI_Op_create(findminnorm, 1, &MPI_MINNORM);

  for (i=0; i < N; i++)
    {
      input[i] = (-size/2+rank+i)*(i+1);
      printf("On rank %d,  input[%d] = %f\n", rank, i, input[i]);
      output[i] = -1;
    }

  MPI_Allreduce(&input, &output, N, MPI_DOUBLE, MPI_MINNORM, MPI_COMM_WORLD);

  for (i=0; i < N; i++)
    {
      printf("On rank %d, output[%d] = %f\n", rank, i, output[i]);
    }

  MPI_Finalize();
}

The initialisation is a bit random but I think it serves to illustrate the point (although your comparison function should really cope with the situation where all inputs are negative):

mpirun -n 5 ./minnorm | grep input | sort
On rank 0,  input[0] = -2.000000
On rank 0,  input[1] = -2.000000
On rank 1,  input[0] = -1.000000
On rank 1,  input[1] = 0.000000
On rank 2,  input[0] = 0.000000
On rank 2,  input[1] = 2.000000
On rank 3,  input[0] = 1.000000
On rank 3,  input[1] = 4.000000
On rank 4,  input[0] = 2.000000
On rank 4,  input[1] = 6.000000

mpirun -n 5 ./minnorm | grep output | sort
On rank 0, output[0] = 1.000000
On rank 0, output[1] = 2.000000
On rank 1, output[0] = 1.000000
On rank 1, output[1] = 2.000000
On rank 2, output[0] = 1.000000
On rank 2, output[1] = 2.000000
On rank 3, output[0] = 1.000000
On rank 3, output[1] = 2.000000
On rank 4, output[0] = 1.000000
On rank 4, output[1] = 2.000000

inoutvec is not common for all processes.

Your operation only have to compute inoutvec = min(invec, inoutvec) and the MPI library will take care of the communications and invoking your operator with the appropriate inoutvec.

From the MPI standard chapter 5.9.5 page 185:

Advice to implementors. We outline below a naive and inefficient implementation of MPI_REDUCE not supporting the in place option.

MPI_Comm_size(comm, &groupsize);
MPI_Comm_rank(comm, &rank);
if (rank > 0) {
MPI_Recv(tempbuf, count, datatype, rank-1,...);
User_reduce(tempbuf, sendbuf, count, datatype);
}
if (rank < groupsize-1) {
MPI_Send(sendbuf, count, datatype, rank+1, ...);
}
/* answer now resides in process groupsize-1 ... now send to root
*/
if (rank == root) {
MPI_Irecv(recvbuf, count, datatype, groupsize-1,..., &req);
}
if (rank == groupsize-1) {
MPI_Send(sendbuf, count, datatype, root, ...);
}
if (rank == root) {
MPI_Wait(&req, &status);
}