Results of parallel calculations on Fortran + mpi are dependent on being/luck of unsignificant commands in code - multiprocessing

Question

Good day to everyone!
I perform parallel calculations using Fortran + mpi (technical details on compilator, OS are listed in the end of this question) and I have the following problem: the result of calculations is changed dramatically if I add or remove a line in my code, that is absolutely irrelevant for very calculations.
The whole code consists of three parts:
the main program IntColl.f90 containing parallel fragments of the code
module m.f90 containing lots of functions to perform specific calculations. One of functions in this module contains the lines being or lack of which in the code is reflected on results of calculations.
module Constants.f90 containing various constants.
The code contains about 2k lines so I only can to show some fragments, which in my opinion have the most significant mean to the question.
Here are code snippets from the main program file IntColl.f90 with parallel calculations:
DO counter1 = 1, NG
IF ( MOD ( counter1, CN ) .EQ. RANK ) THEN
YDOT ( counter1 ) = FACTOR * ABt ( dmgw ( counter1 ) ) + &
FACTOR1 * CIng ( dmgw ( counter1 ) )
YDOT ( counter1 + NG ) = FACTOR * ABMt ( dmgw ( counter1 ) )
END IF
END DO
DO counter = 1, ( CN - 1 )
IF ( RANK .EQ. counter ) THEN
CALL MPI_SEND ( &
YDOT ( counter ), &
1, &
MPI_DP_VEC, &
0, &
1, &
MPI_COMM_WORLD, &
IERROR &
)
END IF
IF ( RANK .EQ. 0 ) THEN
CALL MPI_RECV ( &
YDOT ( counter ), &
1, &
MPI_DP_VEC, &
counter, &
1, &
MPI_COMM_WORLD, &
MPI_STATUS_IGNORE, &
IERROR &
)
END IF
END DO
CALL MPI_BCAST ( &
YDOT ( 1 ), &
2 * NG, &
MPI_DOUBLE_PRECISION, &
RANK, &
MPI_COMM_WORLD, &
IERROR &
)
CN is the number of threads.
Next, here is the function CIng ( p ) from module m.f90 called within parallel piece of the code:
REAL ( DP ) FUNCTION CIng ( p ) ! Collisional integral for \nu + \gamma
IMPLICIT NONE
INTEGER :: i, CTR
REAL ( DP ) :: p ! p is coordinate momentum of initial neutrino: nu ( p ) + gamma ( k ) --> nu ( q ) + e^- ( q1 ) + e^+ ( q2 )
REAL ( DP ) :: const_A
REAL ( DP ) :: const_i
REAL ( DP ) :: const
REAL ( DP ) :: a, b
REAL ( DP ) :: z1, z2, z3, z4, z5, z6, z7
i = 0
CTR = 0
const_A = 128.D+00 * PI * FSC * G_F ** 2 * ( M_E / X ) ** 2
const_i = ( DSQRT ( YMAX ** 2 + X ** 2 ) - X ) ** 2 * YMAX / 2.D+00 ** 8 / PI ** 5 / p * ( M_E / X ) ** 3
const = const_A * const_i
a = 0.D+00
z1 = 0.D+00
z2 = 0.D+00
z3 = 0.D+00
z4 = 0.D+00
z5 = 0.D+00
z6 = 0.D+00
z7 = 0.D+00
DO i = 1, NS
b = 0.D+00
z1 = x1 ( i )
z2 = x2 ( i )
z3 = x3 ( i )
z4 = x4 ( i )
z5 = x5 ( i )
z6 = x6 ( i )
z7 = x7 ( i )
b = UndInt ( p, z1, z2, z3, z4, z5, z6, z7 )
a = a + b
IF ( b .NE. 0.D+00 ) THEN
CTR = CTR + 1
WRITE ( 9, "(2ES23.15)" ) x2 ( i ), x3 ( i ) ! This #1
END IF
END DO
print *, p, CTR, 'q' ! This #2
CIng = const * a / NS
END FUNCTION CIng
The thing is when I comment lines marked with labels This #1 and This #2 the result of calculations I obtain differs catastrophically from the results obtained when these lines are not commented. Note: results of calculations obtained are "good" in my opinion if at least one of these two lines are not commented.
Some clarification: the result of my calculations is some physical dependence which should be at least piecewise smooth. I really obtain points which lie on a piecewise smooth curve at some conditions (third level of optimization at compilation and not commented line This #1), and such result I guess "good". When I comment I/O instructions points become similar to noise without any visual dependence. And I consider intuitively such the result "bad".
The function CIng ( p ) calls one more function UndInt ( p, z1, z2, z3, z4, z5 ,z6, z7 ) which in turn calls next functions, but all these functions just perform some calculations.
Arrays x1 ( i ), x2 ( i ), ... are filled with numbers from a file in IntColl.f90. These arrays are declared in the module Constants.f90 and this module are used in the main program IntColl.f90 and in the module m.f90.
I have to say that the result of calculations is also dependent on level 'n' of optimization at compilation (-O<n>). When both lines #This 1 and #This 2 are commented results of calculations are bad at any level of optimization.
I tried to add options -Wall and -Wextra when compiling. These options showed lots of warnings of following types:
<variable> may be uninitialized in this function
and
Inequality comparison for REAL(8) at (1).
Adding of options -fbacktrace -fcheck=all -g when compiling lead to "bad" result even if one of lines This #1 and This #2 are not commented.
The major part of these warnings are related to functions from fortran libraries I use in my program (e.g., DLSODA).
I also want to specify that time of calculation of value CIng ( dmgw ( counter1 ) ) grows monotonously when counter1 goes from 1 to NG (NG is 96 if it matters).
Here are a few technical details:
The code is written on Fortran90, I compile it using the command
mpif90.mpich *.f90 *.f -O3 -o <name>
and I execute it using the command
mpirun -np <number of threads> ./<name>,
in terminal of my computer with the OS Ubuntu 18.04.04 LTS, version of mpirun is 3.3a2;
on the version of compiler: mpif90 --version reports GNU Fortran (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0.
Please, say me if it is needed any additional information.
I will be grateful for any answer/comment on this qestion or for information about same problems known earlier.
UPD
I am very ashamed, but it has been found out that my problem is absolutely not related with multiprocessing.
The thing is I had in program a variable that could stay uninitialized when program is executed.
Namely, in module m.f90 I have a function UndInt_ ( ... ) containing internal variables a and b.
If these variables are not initialized immediately after their declaration their initialization is performed within a chain of quite intricate nested IF conditions:
REAL ( DP ) FUNCTION UndInt_ ( p, k, eps1, eps2, cosTh1, cosTh2, phi1, phi2 )
IMPLICIT NONE
REAL ( DP ), INTENT ( IN ) :: p, k, eps1, eps2, cosTh1, cosTh2, phi1, phi2
REAL ( DP ) :: q1 ! q1 is the coordinate momentum of electron ( the momentum of electron multiplied by ( X / M_E ) )
REAL ( DP ) :: q2 ! q2 is the coordinate momentum of positron
REAL ( DP ) :: sinTh1
REAL ( DP ) :: sinTh2
REAL ( DP ) :: cosph1
REAL ( DP ) :: cosph2
REAL ( DP ) :: cosThk ! cos of the polar angle of electron
REAL ( DP ) :: q ! the absolute value of coordinate momentum of outgoing neutrino
REAL ( DP ) :: a, b
a = 0.0 ! This variable
b = 0.0 ! And this variable
q1 = DSQRT ( eps1 ** 2 - X ** 2 )
q2 = DSQRT ( eps2 ** 2 - X ** 2 )
sinTh1 = DSQRT ( 1.D+00 - cosTh1 ** 2 )
sinTh2 = DSQRT ( 1.D+00 - cosTh2 ** 2 )
cosph1 = DCOS ( phi1 )
cosph2 = DCOS ( phi2 )
IF ( ( k ** 2 * ( q1 * cosTh1 + q2 * cosTh2 - p ) ** 2 + &
k ** 2 * ( q1 * sinTh1 * cosph1 + q2 * sinTh2 * cosph2 ) ** 2 &
) .LT. &
( p * ( k - eps1 - eps2 ) - k * ( eps1 + eps2 ) + eps1 * eps2 + X ** 2 + &
p * q1 * cosTh1 + p * q2 * cosTh2 - &
q1 * q2 * ( sinTh1 * sinTh2 * DCOS ( phi1 - phi2 ) + cosTh1 * cosTh2 ) &
) ** 2 &
) THEN
a = 0.0
b = 0.0
ELSE
IF ( ( cosThk1 ( p, k, eps1, eps2, cosTh1, cosTh2, phi1, phi2 ) .GT. 1.D+00 ) &
.OR. ( cosThk1 ( p, k, eps1, eps2, cosTh1, cosTh2, phi1, phi2 ) .LT. -1.D+00 ) &
.AND. ( cosThk2 ( p, k, eps1, eps2, cosTh1, cosTh2, phi1, phi2 ) .GT. 1.D+00 ) &
.OR. ( cosThk2 ( p, k, eps1, eps2, cosTh1, cosTh2, phi1, phi2 ) .LT. -1.D+00 ) &
) THEN
a = 0.0
b = 0.0
ELSE
IF ( ( DABS ( p + k - eps1 - eps2 - &
q_1 ( p, k, eps1, eps2, cosTh1, cosTh2, phi1, phi2 ) &
) / &
( p + k - eps1 - eps2 ) &
) .LT. 1.D-015 &
) THEN
cosThk = cosThk1 ( p, k, eps1, eps2, cosTh1, cosTh2, phi1, phi2 )
q = q_1 ( p, k, eps1, eps2, cosTh1, cosTh2, phi1, phi2 )
a = UndInt1 ( p, k, q, eps1, eps2, cosThk, cosTh1, cosTh2, phi1, phi2 )
END IF
IF ( ( DABS ( p + k - eps1 - eps2 - &
q_2 ( p, k, eps1, eps2, cosTh1, cosTh2, phi1, phi2 ) &
) / &
( p + k - eps1 - eps2 ) &
) .LT. 1.D-015 .AND. &
( DABS ( &
( cosThk1 ( p, k, eps1, eps2, cosTh1, cosTh2, phi1, phi2 ) - &
cosThk2 ( p, k, eps1, eps2, cosTh1, cosTh2, phi1, phi2 ) ) / &
cosThk1 ( p, k, eps1, eps2, cosTh1, cosTh2, phi1, phi2 ) &
) &
) .GT. 1.D-015 &
) THEN
cosThk = cosThk2 ( p, k, eps1, eps2, cosTh1, cosTh2, phi1, phi2 )
q = q_2 ( p, k, eps1, eps2, cosTh1, cosTh2, phi1, phi2 )
b = UndInt1 ( p, k, q, eps1, eps2, cosThk, cosTh1, cosTh2, phi1, phi2 )
END IF
END IF
END IF
UndInt_ = a + b
END FUNCTION UndInt_
It looks like for some cases these variables has been stayed uninitialized within the function UndInt_ and were "dirty".
Finally a value of variable UndInt_ that is returned by finction UndInt_ ( ... ) is defined with variables a and b, so a junk from these variables has been passed to variable UndInt_.
That's it.
I understand, that this bug could be fixed immediately if I used options -fbacktrace -fcheck=all -g at compilation.
I would like to say thank you one more time for answers and comments.
I am going to be more careful in future.