gsl_integration_qag works with 1 core (with/without openMP), but fails with multi-threads (i.e. >1).
Some information that may help...
gsl-2.5
#define _OPENMP 201107
Depending on the number of cores, I can get error reports of:
gsl: qag.c:248: ERROR: roundoff error prevents tolerance from being achieved (comment: usually with a small number of cores)
gsl: qag.c:257: ERROR: maximum number of subdivisions reached (comment: usually with a large number of cores)
A large max iteration number given to gsl_integration_qag only delays the code to crash.
The integration function is (can be more specific if needed):
double Func(double Param1, ..., double ParamN){
double result, error;
gsl_function F;
gsl_integration_workspace * w
= gsl_integration_workspace_alloc (1000);
struct parameters_gsl_int_ parameters_gsl = {
.Param1 = Param1,
...
.ParamN = ParamN,};
F.function = &func_integrand;
F.params = ¶meters_gsl;
gsl_integration_qag (&F, LOWER_LIMIT, UPPER_LIMIT, 0, 0.001,
1000, GSL_INTEG_GAUSS61, w, &result, &error);
gsl_integration_workspace_free (w);
return result;
}
The OpenMP part that calls the integration is:
void call_Func(int Nbin, double array[], double Param1[], double Param2, ... double ParamN){
int i;
...
#pragma omp parallel shared(Nbin, array, Param1, ..., ParamN) private(i)
{
#pragma omp for
for (i=0; i<Nbin; i++)
array[i] = Func(Param1[i], Param2, ..., ParamN);
}
...
}
I'm new to both GSL and openMP. I hope I am using gsl_integration_qag correctly and the definition of shared or private variables makes sense.
btw, it's the same question as this 2014 one (gsl openmp failed integration), but I couldn't find the solution in this post.
Problem solved...
It is actually due to func_integrand having also a term which is estimated using gsl_integration_qag. There were some global variables adopted in this calculation, which I didn't capture before.
Related
// return 1 if in set, 0 otherwise
int inset(double real, double img, int maxiter){
double z_real = real;
double z_img = img;
for(int iters = 0; iters < maxiter; iters++){
double z2_real = z_real*z_real-z_img*z_img;
double z2_img = 2.0*z_real*z_img;
z_real = z2_real + real;
z_img = z2_img + img;
if(z_real*z_real + z_img*z_img > 4.0) return 0;
}
return 1;
}
// count the number of points in the set, within the region
int mandelbrotSetCount(double real_lower, double real_upper, double img_lower, double img_upper, int num, int maxiter){
int count=0;
double real_step = (real_upper-real_lower)/num;
double img_step = (img_upper-img_lower)/num;
for(int real=0; real<=num; real++){
for(int img=0; img<=num; img++){
count+=inset(real_lower+real*real_step,img_lower+img*img_step,maxiter);
}
}
return count;
}
// main
int main(int argc, char *argv[]){
double real_lower;
double real_upper;
double img_lower;
double img_upper;
int num;
int maxiter;
int num_regions = (argc-1)/6;
for(int region=0;region<num_regions;region++){
// scan the arguments
sscanf(argv[region*6+1],"%lf",&real_lower);
sscanf(argv[region*6+2],"%lf",&real_upper);
sscanf(argv[region*6+3],"%lf",&img_lower);
sscanf(argv[region*6+4],"%lf",&img_upper);
sscanf(argv[region*6+5],"%i",&num);
sscanf(argv[region*6+6],"%i",&maxiter);
printf("%d\n",mandelbrotSetCount(real_lower,real_upper,img_lower,img_upper,num,maxiter));
}
return EXIT_SUCCESS;
}
I need to convert the above code into openMP. I know how to do it for a single matrix or image but i have to do it for 2 images at the same time
the arguments are as follows
$./mandelbrot -2.0 1.0 -1.0 1.0 100 10000 -1 1.0 0.0 1.0 100 10000
Any suggestion how to divide the work in to different threads for the two images and then further divide work for each image.
thanks in advance
If you want to process multiple images at a time, you need to add a #pragma omp parallel for into the loop in the main body such as:
#pragma omp parallel for private(real_lower, real_upper, img_lower, img_upper, num, maxiter)
for(int region=0;region<num_regions;region++){
// scan the arguments
sscanf(argv[region*6+1],"%lf",&real_lower);
sscanf(argv[region*6+2],"%lf",&real_upper);
sscanf(argv[region*6+3],"%lf",&img_lower);
sscanf(argv[region*6+4],"%lf",&img_upper);
sscanf(argv[region*6+5],"%i",&num);
sscanf(argv[region*6+6],"%i",&maxiter);
printf("%d\n",mandelbrotSetCount(real_lower,real_upper,img_lower,img_upper,num,maxiter));
}
Notice that some variables need to be classified as private (i.e. each thread has its own copy).
Now, if you want additional parallelism you need nested OpenMP (see nested and NESTED_OMP in OpenMP specification) as the work will be spawned by OpenMP threads -- but note that nesting may not give you a performance boost always.
In this case, what about adding a #pragma omp parallel for (with the appropriate reduction clause so that each thread accumulates into count) into the mandelbrotSetCount routine such as
// count the number of points in the set, within the region
int mandelbrotSetCount(double real_lower, double real_upper, double img_lower, double img_upper, int num, int maxiter)
{
int count=0;
double real_step = (real_upper-real_lower)/num;
double img_step = (img_upper-img_lower)/num;
#pragma omp parallel for reduction(+:count)
for(int real=0; real<=num; real++){
for(int img=0; img<=num; img++){
count+=inset(real_lower+real*real_step,img_lower+img*img_step,maxiter);
}
}
return count;
}
The whole approach would split images between threads first and then the rest of the available threads would be able to split the loop iterations in this routine among all the available threads each time you invoke the routine.
EDIT
As user Hristo suggest's on the comments, the mandelBrotSetCount routine might be unbalanced (the best reason is that the user simply requests a different number of maxiter) on each invocation. One way to address this performance issue might be to use dynamic thread scheduling in the routine. So rather than having
#pragma omp parallel for reduction(+:count)
we might want to have
#pragma omp parallel for reduction(+:count) schedule(dynamic,N)
and here N should be a relatively small value (and likely larger than 1).
I'm a learning Cuda student, and I would like to optimize the execution time of my kernel function. As a result, I realized a short program computing the difference between two pictures. So I compared the execution time between a classic CPU execution in C, and a GPU execution in Cuda C.
Here you can find the code I'm talking about:
int *imgresult_data = (int *) malloc(width*height*sizeof(int));
int size = width*height;
switch(computing_type)
{
case GPU:
HANDLE_ERROR(cudaMalloc((void**)&dev_data1, size*sizeof(unsigned char)));
HANDLE_ERROR(cudaMalloc((void**)&dev_data2, size*sizeof(unsigned char)));
HANDLE_ERROR(cudaMalloc((void**)&dev_data_res, size*sizeof(int)));
HANDLE_ERROR(cudaMemcpy(dev_data1, img1_data, size*sizeof(unsigned char), cudaMemcpyHostToDevice));
HANDLE_ERROR(cudaMemcpy(dev_data2, img2_data, size*sizeof(unsigned char), cudaMemcpyHostToDevice));
HANDLE_ERROR(cudaMemcpy(dev_data_res, imgresult_data, size*sizeof(int), cudaMemcpyHostToDevice));
float time;
cudaEvent_t start, stop;
HANDLE_ERROR( cudaEventCreate(&start) );
HANDLE_ERROR( cudaEventCreate(&stop) );
HANDLE_ERROR( cudaEventRecord(start, 0) );
for(int m = 0; m < nb_loops ; m++)
{
diff<<<height, width>>>(dev_data1, dev_data2, dev_data_res);
}
HANDLE_ERROR( cudaEventRecord(stop, 0) );
HANDLE_ERROR( cudaEventSynchronize(stop) );
HANDLE_ERROR( cudaEventElapsedTime(&time, start, stop) );
HANDLE_ERROR(cudaMemcpy(imgresult_data, dev_data_res, size*sizeof(int), cudaMemcpyDeviceToHost));
printf("Time to generate: %4.4f ms \n", time/nb_loops);
break;
case CPU:
clock_t begin = clock(), diff;
for (int z=0; z<nb_loops; z++)
{
// Apply the difference between 2 images
for (int i = 0; i < height; i++)
{
tmp = i*imgresult_pitch;
for (int j = 0; j < width; j++)
{
imgresult_data[j + tmp] = (int) img2_data[j + tmp] - (int) img1_data[j + tmp];
}
}
}
diff = clock() - begin;
float msec = diff*1000/CLOCKS_PER_SEC;
msec = msec/nb_loops;
printf("Time taken %4.4f milliseconds", msec);
break;
}
And here is my kernel function:
__global__ void diff(unsigned char *data1 ,unsigned char *data2, int *data_res)
{
int row = blockIdx.x;
int col = threadIdx.x;
int v = col + row*blockDim.x;
if (row < MAX_H && col < MAX_W)
{
data_res[v] = (int) data2[v] - (int) data1[v];
}
}
I obtained these execution time for each one
CPU: 1,3210ms
GPU: 0,3229ms
I wonder why GPU result is not as lower as it should be. I am a beginner in Cuda so please be comprehensive if there are some classic errors.
EDIT1:
Thank you for your feedback. I tried to delete the 'if' condition from the kernel but it didn't change deeply my program execution time.
However, after having install Cuda profiler, it told me that my threads weren't running concurrently. I don't understand why I have this kind of message, but it seems true because I only have a 5 or 6 times faster application with GPU than with CPU. This ratio should be greater, because each thread is supposed to process one pixel concurrently to all the other ones. If you have an idea of what I am doing wrong, it would be hepful...
Flow.
Here are two things you could do which may improve the performance of your diff kernel:
1. Let each thread do more work
In your kernel, each thread handles just a single element; but having a thread do anything already has a bunch of overhead, at the block and the thread level, including obtaining the parameters, checking the condition and doing address arithmetic. Now, you could say "Oh, but the reads and writes take much more time then that; this overhead is negligible" - but you would be ignoring the fact, that the latency of these reads and writes is hidden by the presence of many other warps which may be scheduled to do their work.
So, let each thread process more than a single element. Say, 4, as each thread can easily read 4 bytes at once into a register. Or even 8 or 16; experiment with it. Of course you'll need to adjust your grid and block parameters accordingly.
2. "Restrict" your pointers
__restrict is not part of C++, but it is supported in CUDA. It tells the compiler that accesses through different pointers passed to the function never overlap. See:
What does the restrict keyword mean in C++?
Realistic usage of the C99 'restrict' keyword?
Using it allows the CUDA compiler to apply additional optimizations, e.g. loading or storing data via non-coherent cache. Indeed, this happens with your kernel although I haven't measured the effects.
3. Consider using a "SIMD" instruction
CUDA offers this intrinsic:
__device__ unsigned int __vsubss4 ( unsigned int a, unsigned int b )
Which subtracts each signed byte value in a from its corresponding one in b. If you can "live" with the result, rather than expecting a larger int variable, that could save you some of work - and go very well with increasing the number of elements per thread. In fact, it might let you increase it even further to get to the optimum.
I don't think you are measuring times correctly, memory copy is a time consuming step in GPU that you should take into account when measuring your time.
I see some details that you can test:
I suppose you are using MAX_H and MAX_H as constants, you may consider doing so using cudaMemcpyToSymbol().
Remember to sync your threads using __syncthreads(), so you don't get issues between each loop iteration.
CUDA works with warps, so block and number of threads per block work better as multiples of 8, but not larger than 512 threads per block unless your hardware supports it. Here is an example using 128 threads per block: <<<(cols*rows+127)/128,128>>>.
Remember as well to free your allocated memory in GPU and destroying your time events created.
In your kernel function you can have a single variable int v = threadIdx.x + blockIdx.x * blockDim.x .
Have you tested, beside the execution time, that your result is correct? I think you should use cudaMallocPitch() and cudaMemcpy2D() while working with arrays due to padding.
Probably there are other issues with the code, but here's what I see. The following lines in __global__ void diff are considered not optimal:
if (row < MAX_H && col < MAX_W)
{
data_res[v] = (int) data2[v] - (int) data1[v];
}
Conditional operators inside a kernel result in warp divergence. It means that if and else parts inside a warp are executed in sequence, not in parallel. Also, as you might have realized, if evaluates to false only at borders. To avoid the divergence and needless computation, split your image in two parts:
Central part where row < MAX_H && col < MAX_W is always true. Create an additional kernel for this area. if is unnecessary here.
Border areas that will use your diff kernel.
Obviously you'll have modify your code that calls the kernels.
And on a separate note:
GPU has throughput-oriented architecture, but not latency-oriented as CPU. It means CPU may be faster then CUDA when it comes to processing small amounts of data. Have you tried using large data sets?
CUDA Profiler is a very handy tool that will tell you're not optimal in the code.
My SSE code works completely fine on Windows platform, but when I run this on Linux I am facing many issues. One amongst them is this:
It's just a sample illustration of my code:
int main(int ref, int ref_two)
{
__128i a, b;
a.m128i_u8[0] = ref;
b.m128i_u8[0] = ref_two;
.
.
.
.....
}
Error 1:
error : request for member 'm128i_u8' in something not a structure or union
In this thread it gives the solution of to use appropriate _mm_set_XXX intrinsics instead of the above method as it only works on Microsoft.
SSE intrinsics compiling MSDN code with GCC error?
I tried the above method mentioned in the thread I have replaced set instruction in my program but it is seriously affecting the performance of my application.
My code is massive and it needs to be changed at 2000 places. So I am looking for better alternative without affecting the performance of my app.
Recently I got this link from Intel, which says to use -fms-diaelect option to port it from windows to Linux.
http://software.intel.com/sites/products/documentation/doclib/iss/2013/compiler/cpp-lin/GUID-7A69898B-BDBB-4AA9-9820-E4A590945903.htm
Has anybody tried the above method? Has anybody found the solution to porting large code to Linux?
#Paul, here is my code and I have placed a timer to measure the time taken by both methods and the results were shocking.
Code 1: 115 ms (Microsoft method to access elements directly)
Code 2: 151 ms (using set instruction)
It costed me a 36 ms when i used set in my code.
NOTE: If I replace in single instruction of mine it takes 36 ms and imagine the performance degrade which I am going to get if I replace it 2000 times in my program.
That's the reason I am looking for a better alternative other than set instruction
Code 1:
__m128i array;
unsigned char* temp_src;
unsigned char* temp_dst;
for (i=0; i< 20; i++)
{
for (j=0; j< 1600; j+= 16)
{
Timerstart(&x);
array = _mm_loadu_si128 ((__m128i *)(src));
array.m128i_u8[0] = 36;
y+ = Timerstop(x);
_mm_store_si128( (__m128i *)temp_dst,array);
}
}
Code2:
__m128i array;
unsigned char* temp_src;
unsigned char* temp_dst;
for (i=0; i< 20; i++)
{
for (j=0; j< 1600; j+= 16)
{
Timerstart(&x);
array = _mm_set_epi8(*(src+15),*(src+14),*(src+13),*(src+12),
*(src+11),*(src+10),*(src+9), *(src+8),
*(src+7), *(src+6), *(src+5), *(src+4),
*(src+3), *(src+2), *(src+1), 36 );
y+ = Timerstop(x);
_mm_store_si128( (__m128i *)temp_dst,array);
}
}
You're trying to use a non-portable Microsoft-ism. Just stick to the more portable intrinsics, e.g. _mm_set_epi8:
__128i a = _mm_set_epi8(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ref);
This will work on all platforms and compilers.
If you're seeing performance issues then it's probably because you're doing something inefficient inside a loop - without seeing the actual code though it's not possible to make any specific suggestions on making the code more efficient.
EDIT
Often there are much more efficient ways of loading a vector with a combination of values such as in your example, e.g.:
#include "smmintrin.h" // SSE4.1
for (...)
{
for (...)
{
__m128i v = _mm_loadu_si128(0, (__m128i *)src); // load vector from src..src+15
v = _mm_insert_epi8(v, 0, 36); // replace element 0 with constant `36`
_mm_storeu_si128((__m128i *)dst, v); // store vector at dst..dst+15
}
}
This translates to just 3 instructions. (Note: if you can't assume SSE4.1 minimum then the _mm_insert_epi8 can be replaced with two bitwise intrinsics - this will still be much more efficient than using _mm_set_epi8).
I need some help to parallelize the pi calculation with the monte carlo method with openmp by a given random number generator, which is not thread safe.
First: This SO thread didn't help me.
My own try is the following #pragma omp statements. I thought the i, x and y vars should be init by each thread and should than be private. z ist the sum of all hits in the circle, so it should be summed after the implied barriere after the for loop.
Think the main problem ist the static state var of the random number generator. I made a critical section where the functions are called, so that only one thread per time could execute it. But the Pi solutions doesn't scale with more higher values.
Note: I should not use another RNG, but its okay to make little changes on it.
int main (int argc, char *argv[]) {
int i, z = 0, threads = 8, iters = 100000;
double x,y, pi;
#pragma omp parallel firstprivate(i,x,y) reduction(+:z) num_threads(threads)
for (i=0; i<iters; ++i) {
#pragma omp critical
{
x = rng_doub(1.0);
y = rng_doub(1.0);
}
if ((x*x+y*y) <= 1.0)
z++;
}
pi = ((double) z / (double) (iters*threads))*4.0;
printf("Pi: %lf\n", pi);;
return 0;
}
This RNG is actually an included file, but as I'm not sure if I create the header file correct, I integrated it in the other program file, so I have only one .c file.
#define RNG_MOD 741025
int rng_int(void) {
static int state = 0;
return (state = (1366 * state + 150889) % RNG_MOD);
}
double rng_doub(double range) {
return ((double) rng_int()) / (double) ((RNG_MOD - 1)/range);
}
I've also tried to make the static int state global, but it doesn't change my result, maybe I done it wrong. So please could you help me make the correct changes? Thank you very much!
Your original linear congruent PRNG has a cycle length of 49400, therefore you are only getting 29700 unique test points. This is a terrible generator to be used for any kind of Monte Carlo simulations. Even if you make 100000000 trials, you won't get any closer to the true value of Pi because you are simply repeating the same points over and over again and as a result both the final value of z and iters are simply multiplied by the same constant, which cancel in the end during the division.
The per-thread seed introduced by Z boson improves the situation a little bit with the number of unique points increasing with the total number of OpenMP threads. The increase is not linear since if the seed of one PRNG falls in the sequence of another PRNG, both PRNGs produce the same sequence shifted with no more than 49400 elements. Given the cycle length, each PRNG covers 49400/RNG_MOD = 6,7% of the total output range and that is the probability of two PRNGs being synchronised. There are a total of RNG_MOD/49400 = 15 unique sequences possible. It basically means that in the best seeding case scenario you won't be able to get past 30 threads as any other thread would simply repeat the result of some of the others. The multiplier 2 comes from the fact that each point uses two elements from the sequence and therefore it is possible to get a different set of points if you shift the sequence by one element.
The ultimate solution is to completely drop your PRNG and stick to something like Mersenne twister MT19937, which has a cycle length of 219937 − 1 and a very strong seeding algorithm. If you are not able to use another PRNG as you state in your question, at least modify the constants of the LCG to match those used in rand():
int rng_int(void) {
static int state = 1;
// & 0x7fffffff is equivalent to modulo with RNG_MOD = 2^31
return (state = (state * 1103515245 + 12345) & 0x7fffffff);
}
Note that rand() is not a good PRNG - it is still bad. It is just a little better than the one used in your code.
Try the code below. It makes a private state for each thread. I did something similar with the at rand_r function Why does calculation with OpenMP take 100x more time than with a single thread?
Edit: I updated my code using some of Hristo's suggestions. I used threadprivate (for the first time). I also used a better rand function which gives a better estimate of pi but it's still not good enough.
One strange things was I had to define the function rng_int after threadprivate otherwise I got an error "error: 'state' declared 'threadprivate' after first use". I should probably ask a question about this.
//gcc -O3 -Wall -pedantic -fopenmp main.c
#include <omp.h>
#include <stdio.h>
#define RNG_MOD 0x80000000
int state;
int rng_int(void);
double rng_doub(double range);
int main() {
int i, numIn, n;
double x, y, pi;
n = 1<<30;
numIn = 0;
#pragma omp threadprivate(state)
#pragma omp parallel private(x, y) reduction(+:numIn)
{
state = 25234 + 17 * omp_get_thread_num();
#pragma omp for
for (i = 0; i <= n; i++) {
x = (double)rng_doub(1.0);
y = (double)rng_doub(1.0);
if (x*x + y*y <= 1) numIn++;
}
}
pi = 4.*numIn / n;
printf("asdf pi %f\n", pi);
return 0;
}
int rng_int(void) {
// & 0x7fffffff is equivalent to modulo with RNG_MOD = 2^31
return (state = (state * 1103515245 + 12345) & 0x7fffffff);
}
double rng_doub(double range) {
return ((double)rng_int()) / (((double)RNG_MOD)/range);
}
You can see the results (and edit and run the code) at http://coliru.stacked-crooked.com/a/23c1753a1b7d1b0d
I have two very simple pieces of codes. I am trying to parallel them as follows:
double sk = 0, ed = 0;
#pragma omp parallel shared(Z,Zo,U1,U2,U3) private(i) reduction(+: sk, ed)
{
#pragma omp for
for (i=0;i<imgDim;i++)
{
sk += (Z[i]-Zo[i])*(Z[i]-Zo[i]);
ed += U1[i]*U1[i] + U2[i]*U2[i] + U3[i]*U3[i];
}
}
//////////////////////////////////////////////////////////////////////////////////////
double rk = 0, epri = 0, ex = 0, ez = 0;
#pragma omp parallel shared(X,Z) private(i) reduction(+: rk, ex,ez)
{
#pragma omp for
for(i = 0; i<imgDim; i++)
{
rk += (X[0][i]-Z[i])*(X[0][i]-Z[i]) + (X[1][i]-Z[i])*(X[1][i]-Z[i]) + (X[2][i]-Z[i])*(X[2][i]-Z[i]);
ex += X[0][i]*X[0][i] + X[1][i]*X[1][i] + X[2][i]*X[2][i];
ez += Z[i]*Z[i];
}
}
Z, Zo,U1,U2,U3,X are all big matrices. imgDim is 4 million. The speed up is not as expected. On a 16 core machine, the speed up of these two pieces of small codes is only two times. I do not understand why OMP presents this behavior because these two codes only add something up. This should be what OMP good at.
The more strange behavior is MPI slow things down when I try to parallel these code by using MPI as follows:
int startval = imgDim*pid/np;
int endval = imgDim*(pid+1)/np-1;
int ierr;
double p_sum_sk = 0;
double p_sum_ed = 0;
for (i=startval;i<=endval;i++)
{
sk += (Z[i]-Zo[i])*(Z[i]-Zo[i]);
ed += U1[i]*U1[i] + U2[i]*U2[i] + U3[i]*U3[i];
}
MPI_Reduce(&sk, &p_sum_sk, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
MPI_Reduce(&ed, &p_sum_ed, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
MPI_Bcast(&sk, 1, MPI_DOUBLE, 0, MPI_COMM_WORLD);
MPI_Bcast(&ed, 1, MPI_DOUBLE, 0, MPI_COMM_WORLD);
/////////////////////////////////////////////////////////////////////////////////////
int startval = imgDim*pid/np;
int endval = imgDim*(pid+1)/np-1;
double p_sum_rk = 0.;
double p_sum_ex = 0.;
double p_sum_ez = 0.;
for(i = startval; i<=endval; i++)
{
rk = rk + (X[0][i]-Z[i])*(X[0][i]-Z[i]) + (X[1][i]-Z[i])*(X[1][i]-Z[i]) + (X[2][i]-Z[i])*(X[2][i]-Z[i]);
ex += X[0][i]*X[0][i] + X[1][i]*X[1][i] + X[2][i]*X[2][i];
ez += Z[i]*Z[i];
}
MPI_Reduce(&rk,&p_sum_rk,1,MPI_DOUBLE,MPI_SUM,0,MPI_COMM_WORLD);
MPI_Reduce(&ex,&p_sum_ex,1,MPI_DOUBLE,MPI_SUM,0,MPI_COMM_WORLD);
MPI_Reduce(&ez,&p_sum_ez,1,MPI_DOUBLE,MPI_SUM,0,MPI_COMM_WORLD);
MPI_Bcast(&rk,1,MPI_INT,0,MPI_COMM_WORLD);
MPI_Bcast(&rk,1,MPI_INT,0,MPI_COMM_WORLD);
MPI_Bcast(&epri,1,MPI_INT,0,MPI_COMM_WORLD);
np is the number of processors and pid is the id of current processor. After I use 32 or even 64 processors, it did not show any speed up. It is even slower than the sequential code. I do not understand why. These codes are just adding stuff up. OMP and MPI should be good at it. Can anyone give me a hand?
Your code is memory bound - you load a huge amount of data on each iteration and make simple (i.e. fast) computations over it. If imgDim is 4 million, then even if each element of Z, Zo, U1, U2, U3 is as short as 4 bytes (e.g. they are float or int arrays), their total size would be 80 MiB and this would not fit in the last-level CPU cache even given a dual-socket system. Things would be worse if these arrays hold double values (as hinted by the fact that you reduce into double variables), as it would bump up the memory size twofold. Also, if you use a decent compiler, which is able to vectorise the code (e.g. icc does it by default, GCC requires -ftree-vectorize), even a single thread would be able to saturate the memory bandwidth of the CPU socket and then running with more than one thread would bring no benefit whatsoever.
I would say that the 2x OpenMP speed-up that you observe on a 16-core system comes from the fact that this system has two CPU sockets and is NUMA, i.e. it has a separate memory controller on each socket and hence when running with 16 threads you utilise twice the memory bandwidth of the single socket. This could be verified if you run the code with two threads only, but bind them in a different way: one thread per core on the same socket or one thread per core but on different sockets. In the first case there should be no speed-up while in the second case the speed-up should be about 2x. Binding threads to cores is (yet) implementation dependent - you could take a look at GOMP_CPU_AFFINITY for GCC and KMP_AFFINITY if you happen to use Intel compilers.
The same applies to the MPI case. Now you have processes instead of threads, but the memory bandwidth limitation stays. Things are even worse, as now there is also communication overhead being added and it could exceed the computation time if the problem size is too small (the ratio depends on the network interconnect - it is lower with faster and less latent interconnects like QDR InfiniBand fabric). But with MPI you have access to more CPU sockets and hence to higher total memory bandwidth. You could launch your code with one MPI process per socket to get the best possible performance out of your system. Process binding (or pinning in Intel's terminology) is also important in that case.