The relation between MPI_Comm_split and MPI_Send / MPI_Recv - parallel-processing

Question

Let's say there are 3 processes, This code works fine:
#include <iostream>
#include <mpi.h>
using namespace std;
int main(){
MPI_Init(NULL, NULL);
int rank; MPI_Comm SubWorld; int buf;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
if (rank == 0){
MPI_Comm_split(MPI_COMM_WORLD, rank, rank, &SubWorld);
MPI_Send(&buf, 1, MPI_INT, 1, 55, MPI_COMM_WORLD);
}
else if (rank == 1){
MPI_Comm_split(MPI_COMM_WORLD, rank, rank, &SubWorld);
MPI_Recv(&buf, 1, MPI_INT, 0, 55, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
}
else MPI_Comm_split(MPI_COMM_WORLD, rank, rank, &SubWorld);
cout << "Done" << endl;
MPI_Finalize();
return 0;
}
Outputs "Done" three times as expected.
But this code has a problem (also 3 processes):
#include <iostream>
#include <mpi.h>
using namespace std;
int main(){
MPI_Init(NULL, NULL);
int rank; MPI_Comm SubWorld; int buf;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
if (rank == 0){
MPI_Comm_split(MPI_COMM_WORLD, rank, rank, &SubWorld);
MPI_Send(&buf, 1, MPI_INT, 1, 55, MPI_COMM_WORLD);
}
else if (rank == 1){
MPI_Recv(&buf, 1, MPI_INT, 0, 55, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
MPI_Comm_split(MPI_COMM_WORLD, rank, rank, &SubWorld);
}
else MPI_Comm_split(MPI_COMM_WORLD, rank, rank, &SubWorld);
cout << "Done" << endl;
MPI_Finalize();
return 0;
}
There is no output!!
What exactly is the relation between MPI_Comm_split and MPI_Send / MPI_Recv which cause this problem?

Answer 1

MPI_Comm_split() is a collective operation, which means all MPI tasks from the initial communicator (e.g. MPI_COMM_WORLD here) must invoke it at the same time.
In your example, rank 1 hangs in MPI_Recv(), and hence MPI_Comm_split() on rank 0 cannot complete (and MPI_Send() is never invoked) and hence the deadlock.
You might consider padb in order to visualize the state of a MPI program, it would make it easy to see where stacks are stuck at.