Does MPI_Scatter influence MPI_Bcast? - parallel-processing

Question

I'm sending an integer that triggers termination via MPI_Bcast. The root sets a variable called "running" to zero and sends the BCast. The Bcast seems to complete but I can't see that the value is sent to the other processes. The other processes seem to be waiting for an MPI_Scatter to complete. They shouldn't even be able to arrive here.
I have done much research on MPI_Bcast and from what I understand it should be blocking. This is confusing me since the MPI_Bcast from the root seems to complete even though I can't find the matching (receiving) MPI_Bcasts for the other processes. I have surrounded all of my MPI_Bcasts with printfs and the output of those printfs 1) print and 2) print the correct values from the root.
The root looks as follows:
while (running || ...) {
/*Do stuff*/
if (...) {
running = 0;
printf("Running = %d and Bcast from root\n", running);
MPI_Bcast(&running, 1, MPI_INT, 0, MPI_COMM_WORLD);
printf("Root 0 Bcast complete. Running %d\n", running);
/* Do some more stuff and eventually reach Finalize */
printf("Root is Finalizing\n");
MPI_Finalize();
}
}
The other processes have the following code:
while (running) {
doThisFunction(rank);
printf("Waiting on BCast from root with myRank: %d\n", rank);
MPI_Bcast(&running, 1, MPI_INT, 0, MPI_COMM_WORLD);
printf("P%d received running = %d\n", rank, running);
if (running == 0) { // just to make sure.
break;
}
}
MPI_Finalize();
I also have the following in the function "doThisFunction()". This is where the processes seem to be waiting for process 0:
int doThisFunction(...) {
/*Do stuff*/
printf("P%d waiting on Scatter\n", rank);
MPI_Scatter(buffer, 130, MPI_BYTE, encoded, 130, MPI_BYTE, 0, MPI_COMM_WORLD);
printf("P%d done with Scatter\n", rank);
/*Do stuff*/
printf("P%d waiting on gather\n", rank);
MPI_Gather(encoded, 1, MPI_INT, buffer, 1, MPI_INT, 0, MPI_COMM_WORLD);
printf("P%d done with gater\n", rank);
/*Do Stuff*/
return aValue;
}
The output in the command line looks as follows:
P0 waiting on Scatter
P0 done with Scatter
P0 waiting on gather
P0 done with gather
Waiting on BCast from root with myRank: 1
P1 received running = 1
P1 waiting on Scatter
P0 waiting on Scatter
P0 done with Scatter
P0 waiting on gather
P0 done with gather
P1 done with Scatter
P1 waiting on gather
P1 done with gather
Waiting on BCast from root with myRank: 1
P1 received running = 1
P1 waiting on Scatter
Running = 0 and Bcast from root
Root 0 Bcast complete. Running 0
/* Why does it say the Bcast is complete
/* even though P1 didn't output that it received it?
Root is Finalizing
/* Deadlocked...
I'm expecting that P1 receives running as zero and then goes into MPI_Finalize() but rather it gets stuck at the scatter which will not be accessed by the root which is already trying to finalize.
In actuality, the program is in deadlock and won't terminate MPI.
I doubt that the problem is that the scatter is accepting the Bcast value because this doesn't even make sense since the root doesn't call scatter.
Does anyone please have any tips on how to resolve this problem?
Your help is greatly appreciated.

Answer 1

Why does it say the Bcast is complete even though P1 didn't output that it received it?
Note the following definitions from the MPI Standard:
Collective operations can (but are not required to) complete as soon as the caller's participation in the collective communication is finished. ... The completion of a collective operation indicates that the caller is free to modify locations in the communication buffer. It does not indicate that other processes in the group have completed or even started the operation (unless otherwise implied by the description of the operation). Thus, a collective communication operation may, or may not, have the effect of synchronizing all calling processes. This statement excludes, of course, the barrier operation.
According to this definition, your MPI_Bcast on the root process can finish even if there is no MPI_Bcast called by slaves.
(For point-to-point operations, we have different communication modes, such as the synchronous one, to address these issues. Unfortunately, there is no synchronous mode for collectives.)
There seems to be some problem in your code with the order of operations. The root called MPI_Bcast, but process #1 did not and was waiting on MPI_Scatter as your log output indicates.