Proper implementation of triangular solver for sparse matrix with OpenACC - solver

Question

I'm currently implementing triangular solver for sparse matrix and I've try to accelerate then using OpenACC directives. Given my matrix factors LU in sparse CSR format, OpenACC have managed to solve the L factor properly but the U factor gives completely wrong when compared with the true solution of the application. Here is the code of the accelerated kernel for the backward substitution task:
#pragma acc kernels deviceptr( ia, ja, factorValsU, y, x )
{
for ( int idx = size; idx > 0; idx-- )
{
double temp = 0.0;
int rowInit = ia[ idx - 1];
int rowEnd = ia[ idx ];
#pragma acc loop vector reduction( + : temp)
for ( int k = rowInit + 1; k < rowEnd; k++ )
{
temp += factorValsU[ k ] * x[ ja[ k ] ];
}
x[ idx ] = (y[ idx ] - temp) / factorValsU[ rowInit ];
}
}
I'm clueless about why this kernels does produce incorrect result. I've already tried a different version for the kernel where the matrix is saved backwards, i.e from down to top, which in principle could be solved with the follow kernel:
#pragma acc kernels deviceptr( ia, ja, factorValsU, y, x )
{
for ( int idx = 0; idx < size; idx++ )
{
double temp = 0.0;
int rowInit = ia[ idx ];
int rowEnd = ia[ idx + 1 ];
#pragma acc loop vector reduction( + : temp)
for ( int k = rowInit + 1; k < rowEnd; k++ )
{
temp += factorValsU[ k ] * x[ ja[ k ] ];
}
x[ size - idx ] = (y[ size - idx ] - temp) / factorValsU[ rowInit ];
}
}
But the result is always wrong. Did I miss something fundamental about decorating regular code with OpenACC directives to achieve proper results?
As mentioned before, the forward substitution of the L factor is properly working so, for completeness I do post the code here.
#pragma acc kernels deviceptr( ia, ja, factorValsL, y, x )
{
for ( int idx = 0; idx < size; idx++ )
{
double temp = 0.0;
int rowInit = ia[ idx ];
int rowEnd = ia[ idx + 1 ];
#pragma acc loop vector reduction( + : temp)
for ( int k = rowInit; k < rowEnd; k++ )
{
temp += factorValsL[ k ] * x[ ja[ k ] ];
}
x[ idx ] = y[ idx ] - temp;
}
}
Note the subtle difference between the kernel for the forward substitution (works) and backward substitution (both not working), is memory region where the result are saved:
x[ idx ] = y[ idx ] - temp for the L factor
x[ size - idx ] = (y[ size - idx ] - temp) / factorValsU[ rowInit ] for the U factor;
Is there some reason for the U factor solver to compute mistaken results cause of the order in which the assignation (and lecture) in memory is made?.
For completeness, the information provided by the pgi18.4 compiler about the kernel is:
triangularSolverU_acc(int, const int *, const int *, const double *, const double *, double *, bool):
614, Complex loop carried dependence of y->,x->,factorVals-> prevents parallelization
Loop carried dependence of x-> prevents parallelization
Loop carried backward dependence of x-> prevents vectorization
Accelerator kernel generated
Generating Tesla code
614, #pragma acc loop seq
621, #pragma acc loop vector(128) /* threadIdx.x */
Generating reduction(+:temp)
621, Loop is parallelizable
Which shows that the external loop have been serialized and the inner loop is a reduction.

Answer 1

With "kernels", the compiler must prove that the loop does not contain any dependencies and therefor is safe to parallelize. However since your code contains pointers, and that pointers may be aliased to the same memory, the compiler can't prove this so does the safe thing and make the loop run sequential. To override the compiler analysis, you can add a "#pragma acc loop independent" before the outer for loop. The "independent" is an assertion to the compiler that the loop is safe to parallelize. Alternatively, you can use the "parallel loop" directive in place of "kernels" since "parallel" implies "independent".
For the wrong answers, these are often due to the data not properly being synchronized between the host and device copies. How are you managing the data movement? Since you're using "deviceptr", this implies that you're using CUDA.
Also, if you can post a complete reproducing example, it would be easier to help determine the issue.