Strange slow down with MPI and OMP - parallel-processing

Question

I have two very simple pieces of codes. I am trying to parallel them as follows:
double sk = 0, ed = 0;
#pragma omp parallel shared(Z,Zo,U1,U2,U3) private(i) reduction(+: sk, ed)
{
#pragma omp for
for (i=0;i<imgDim;i++)
{
sk += (Z[i]-Zo[i])*(Z[i]-Zo[i]);
ed += U1[i]*U1[i] + U2[i]*U2[i] + U3[i]*U3[i];
}
}
//////////////////////////////////////////////////////////////////////////////////////
double rk = 0, epri = 0, ex = 0, ez = 0;
#pragma omp parallel shared(X,Z) private(i) reduction(+: rk, ex,ez)
{
#pragma omp for
for(i = 0; i<imgDim; i++)
{
rk += (X[0][i]-Z[i])*(X[0][i]-Z[i]) + (X[1][i]-Z[i])*(X[1][i]-Z[i]) + (X[2][i]-Z[i])*(X[2][i]-Z[i]);
ex += X[0][i]*X[0][i] + X[1][i]*X[1][i] + X[2][i]*X[2][i];
ez += Z[i]*Z[i];
}
}
Z, Zo,U1,U2,U3,X are all big matrices. imgDim is 4 million. The speed up is not as expected. On a 16 core machine, the speed up of these two pieces of small codes is only two times. I do not understand why OMP presents this behavior because these two codes only add something up. This should be what OMP good at.
The more strange behavior is MPI slow things down when I try to parallel these code by using MPI as follows:
int startval = imgDim*pid/np;
int endval = imgDim*(pid+1)/np-1;
int ierr;
double p_sum_sk = 0;
double p_sum_ed = 0;
for (i=startval;i<=endval;i++)
{
sk += (Z[i]-Zo[i])*(Z[i]-Zo[i]);
ed += U1[i]*U1[i] + U2[i]*U2[i] + U3[i]*U3[i];
}
MPI_Reduce(&sk, &p_sum_sk, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
MPI_Reduce(&ed, &p_sum_ed, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
MPI_Bcast(&sk, 1, MPI_DOUBLE, 0, MPI_COMM_WORLD);
MPI_Bcast(&ed, 1, MPI_DOUBLE, 0, MPI_COMM_WORLD);
/////////////////////////////////////////////////////////////////////////////////////
int startval = imgDim*pid/np;
int endval = imgDim*(pid+1)/np-1;
double p_sum_rk = 0.;
double p_sum_ex = 0.;
double p_sum_ez = 0.;
for(i = startval; i<=endval; i++)
{
rk = rk + (X[0][i]-Z[i])*(X[0][i]-Z[i]) + (X[1][i]-Z[i])*(X[1][i]-Z[i]) + (X[2][i]-Z[i])*(X[2][i]-Z[i]);
ex += X[0][i]*X[0][i] + X[1][i]*X[1][i] + X[2][i]*X[2][i];
ez += Z[i]*Z[i];
}
MPI_Reduce(&rk,&p_sum_rk,1,MPI_DOUBLE,MPI_SUM,0,MPI_COMM_WORLD);
MPI_Reduce(&ex,&p_sum_ex,1,MPI_DOUBLE,MPI_SUM,0,MPI_COMM_WORLD);
MPI_Reduce(&ez,&p_sum_ez,1,MPI_DOUBLE,MPI_SUM,0,MPI_COMM_WORLD);
MPI_Bcast(&rk,1,MPI_INT,0,MPI_COMM_WORLD);
MPI_Bcast(&rk,1,MPI_INT,0,MPI_COMM_WORLD);
MPI_Bcast(&epri,1,MPI_INT,0,MPI_COMM_WORLD);
np is the number of processors and pid is the id of current processor. After I use 32 or even 64 processors, it did not show any speed up. It is even slower than the sequential code. I do not understand why. These codes are just adding stuff up. OMP and MPI should be good at it. Can anyone give me a hand?

Answer 1

Your code is memory bound - you load a huge amount of data on each iteration and make simple (i.e. fast) computations over it. If imgDim is 4 million, then even if each element of Z, Zo, U1, U2, U3 is as short as 4 bytes (e.g. they are float or int arrays), their total size would be 80 MiB and this would not fit in the last-level CPU cache even given a dual-socket system. Things would be worse if these arrays hold double values (as hinted by the fact that you reduce into double variables), as it would bump up the memory size twofold. Also, if you use a decent compiler, which is able to vectorise the code (e.g. icc does it by default, GCC requires -ftree-vectorize), even a single thread would be able to saturate the memory bandwidth of the CPU socket and then running with more than one thread would bring no benefit whatsoever.
I would say that the 2x OpenMP speed-up that you observe on a 16-core system comes from the fact that this system has two CPU sockets and is NUMA, i.e. it has a separate memory controller on each socket and hence when running with 16 threads you utilise twice the memory bandwidth of the single socket. This could be verified if you run the code with two threads only, but bind them in a different way: one thread per core on the same socket or one thread per core but on different sockets. In the first case there should be no speed-up while in the second case the speed-up should be about 2x. Binding threads to cores is (yet) implementation dependent - you could take a look at GOMP_CPU_AFFINITY for GCC and KMP_AFFINITY if you happen to use Intel compilers.
The same applies to the MPI case. Now you have processes instead of threads, but the memory bandwidth limitation stays. Things are even worse, as now there is also communication overhead being added and it could exceed the computation time if the problem size is too small (the ratio depends on the network interconnect - it is lower with faster and less latent interconnects like QDR InfiniBand fabric). But with MPI you have access to more CPU sockets and hence to higher total memory bandwidth. You could launch your code with one MPI process per socket to get the best possible performance out of your system. Process binding (or pinning in Intel's terminology) is also important in that case.