OpenMP FORTRAN issue with privates - openmp

Question

In the following code, when I am passing the variable "aa" as private, the results are getting bad. The code works fine how it is posted, but when I replace the line
!$OMP PARALLEL PRIVATE(iii,iter,y,i,yt) SHARED(bb)
with
!$OMP PARALLEL PRIVATE(aa,iter,y,i,yt) SHARED(bb)
the code is not working properly.
!!!!!!!! module
module common
use iso_fortran_env
implicit none
integer,parameter:: dp=real64
real(dp):: aa,bb
contains
subroutine evolve(y,yevl)
implicit none
integer(dp),parameter:: id=2
real(dp),intent(in):: y(id)
real(dp),intent(out):: yevl(id)
yevl(1)=y(2)+1.d0-aa*y(1)**2
yevl(2)=bb*y(1)
end subroutine evolve
end module common
use common
implicit none
integer(dp):: iii,iter,i
integer(dp),parameter:: id=2
real(dp),allocatable:: y(:),yt(:)
integer(dp):: OMP_GET_THREAD_NUM, IXD
allocate(y(id)); allocate(yt(id)); y=0.d0; yt=0.d0; bb=0.3d0
!$OMP PARALLEL PRIVATE(iii,iter,y,i,yt) SHARED(bb)
IXD=OMP_GET_THREAD_NUM()
!$OMP DO
do iii=1,20000; print*,iii !! EXPECTED THREADS TO BE OF 5000 ITERATIONS EACH
aa=1.d0+dfloat(iii-1)*0.4d0/2000.d0
loop1: do iter=1,10 !! THE INITIAL CONDITION LOOP
call random_number(y)!! RANDOM INITIALIZATION OF THE VARIABLE
loop2: do i=1,70000 !! ITERATION OF THE SYSTEM
call evolve(y,yt)
y=yt
enddo loop2 !! END OF SYSTEM ITERATION
write(IXD+1,*)aa,yt !!! WRITING FILE CORRESPONDING TO EACH THREAD
enddo loop1 !!INITIAL CONDITION ITERATION DONE
enddo
!$OMP ENDDO
!$OMP END PARALLEL
end
What could be the issue? Works fine when I generate "aa" from "iii" but not when I pass it as a private variable. Thanks in advance for any comments or suggestions.

Answer 1

aa is a module variable. Module variables can either be shared (the default) or threadprivate. Example A.32.2f from the OpenMP standard document illustrates that when module variables are accessed in the dynamic scope of a construct, it is unspecified whether the original variable or the private thread copy is being accessed. This is not the case with threadprivate variables as they are always stored in the thread-local storage, no matter if used inside the lexical scope of a parallel region or not.
There are many scenarios for what happens if you declare a module variable to be private and then access it into a subroutine. What is most likely to happen depends on what kind of analysis the compiler does on the code. Some compilers might detect, that the module subroutine is only called inside the parallel region and hence make aa refer to the private copy of each thread. Other compilers might decide to always access the original module variable. On the other hand, if the subroutine gets inlined in the calling subroutine, then it might refer to the same aa that is used in the calling context (e.g. the private version if aa is declared private)
Here is an example of how gfortran handles PRIVATE(iii,aa,iter,y,i,yt) at the default optimisation level:
; aa is declared as a global symbol in the BSS section
.globl __common_MOD_aa
.bss
.align 8
.type __common_MOD_aa, #object
.size __common_MOD_aa, 8
__common_MOD_aa:
.zero 8
; Here is how evolve accesses aa
...
movsd __common_MOD_aa(%rip), %xmm2
...
; Here is how the assignment to aa is done inside the parallel region
...
movsd %xmm0, -72(%rbp)
...
The private aa is implemented as an automatic variable and stored in the stack of the thread, while evolve uses the value of aa from the module. Therefore this operator:
aa=1.d0+dfloat(iii-1)*0.4d0/2000.d0
only alters the value of aa inside the thread, while evolve uses the original value of aa from outside the parallel region.
At the high optimisation level -O3 gfortran inlines evolve into the parallel region and...
...
mulsd __common_MOD_aa(%rip), %xmm2
...
The inlined code also refers to the global value of aa in the module, i.e. the behaviour is consistent between the two optimisation levels.
The same applies to Intel Fortran.
The correct approach is to declare aa to be threadprivate and to not put it in a private clause:
module common
use iso_fortran_env
implicit none
integer,parameter:: dp=real64
real(dp):: aa,bb
!$OMP THREADPRIVATE(aa)
...
!$OMP PARALLEL PRIVATE(iii,iter,y,i,yt) SHARED(bb)
IXD=OMP_GET_THREAD_NUM()
!$OMP DO
do iii=1,20000; print*,iii !! EXPECTED THREADS TO BE OF 5000 ITERATIONS EACH
aa=1.d0+dfloat(iii-1)*0.4d0/2000.d0
...
Now both the parallel region and evolve will use a private to each thread copy of aa. As access to threadprivate variables is usually slower than access to normal private (stack) variables, on 64-bit x86 systems it might make more sense to pass the value of aa as an argument to evolve instead as suggested by #Bálint Aradi.

Answer 2

You should try to carefully analyze your variables, especially to think about which of them would have different values on the different threads at the same time, as those must be declared OMP private. In your case, both variables aa and iii must be OMP private. Variable iii because it is a counter in a loop which is distributed over the threads, and aa because it gets a value which depends on iii.
EDIT: As each thread calls the evolve subroutine itself and evolve is supposed to use the thread specific value of aa (I guess), you should also pass aa to your subroutine instead of using the module variable aa.
The routine should look like:
subroutine evolve(y, aa, yevl)
integer(dp),parameter:: id=2
real(dp),intent(in):: y(id), aa
real(dp),intent(out):: yevl(id)
yevl(1)=y(2)+1.d0-aa*y(1)**2
yevl(2)=bb*y(1)
end subroutine evolve
and the according call in your main program:
call evolve(y, aa, yt)