Gradient of a function in OpenCL - algorithm

Question

I'm playing around a bit with OpenCL and I have a problem which can be simplified as follows.
I'm sure this is a common problem but I cannot find many references or examples that would show me how this is usually done
Suppose for example you have a function (writing in CStyle syntax)
float function(float x1, float x2, float x3, float x4, float x5)
{
return sin(x1) + x1*cos(x2) + x3*exp(-x3) + x4 + x5;
}
I can also implement the gradient of this function as
void functionGradient(float x1, float x2, float x3, float x4, float x5, float gradient[])
{
gradient[0] = cos(x1) + cos(x2);
gradient[1] = -sin(x2);
gradient[2] = exp(-x3) - x3*exp(-x3);
gradient[3] = 1.0f;
gradient[4] = 1.0f;
}
Now I was thinking of implementing an OpenCL C kernel function that would do the same thing, cause I wanted to speed this up. The only way I have in mind to do this is to assign to each workunit a component of the gradient but then I'd need to put a bunch of if statements within the code to figure which workunit is computing what component, which isn't good in general because of divergence.
Therefore here is the question, how is such problem tackled in general? I'm aware for example of Gradient Descent implementations on GPU, see machine learning with backpropagation for example. So I wonder what is generally done to avoid divergence in the code.
Follow up from suggestion
I'm thinking of a possible SIMD compatible implementation as follows:
/*
Pseudo OpenCL-C code
here weight is a 5x5 array containing weights in {0,1} masking the relevant
computation
*/
__kernel void functionGradient(float x1, float x2, float x3, float x4, float x5, __global float* weight,__global* float gradient)
{
size_t threadId = get_global_id(0);
gradient[threadId] =
weight[5*threadId]*(cos(x1) + cos(x2)) +
weight[5*threadId + 1]*(-sin(x2)) +
weight[5*threadId + 2]*(exp(-x3) - x3*exp(x3)) +
weight[5*threadId + 3] + weight[5*threadId + 4];
barrier(CLK_GLOBAL_MEM_FENCE);
}

Answer 1

If your gradient function only has 5 components, it does not make sense to parallelize it in a way that one thread does one component. As you mentioned, GPU parallelization does not work if the mathematical structure of each components is different (multiple instructionsmultiple data, MIMD).
If you would need to compute the 5-dimensional gradient at 100k different coordinates however, then each thread would do all 5 components for each coordinate and parallelization would work efficiently.
In the backpropagation example, you have one gradient function with thousands of dimensions. In this case you would indeed parallelize the gradient function itself such that one thread computes one component of the gradient. However in this case all gradient components have the same mathematical structure (with different weighting factors in global memory), so branching is not required. Each gradient component is the same equation with different numbers (single instruction multiple data, SIMD). GPUs are designed to only handle SIMD; this is also why they are so energy efficient (~30TFLOPs # 300W) compared to CPUs (which can do MIMD, ~2-3TFLOPs # 150W).
Finally, note that backpropagation / neural nets are specifically designed to be SIMD. Not every new algorithm you come across can be parallelize in this manner.
Coming back to your 5-dimensional gradient example: There are ways to make it SIMD-compatible without branching. Specifically bit-maskimg: You would compute 2 cosines (for componet 1 express the sine through cosine) and one exponent and add all the terms up with a factor in front of each. The terms that you don't need, you multiply by a factor 0. Lastly, the factors are functions of the component ID. However as mentioned above, this only makes sense if you have many thousands to millions of dimensions.
Edit: here the SIMD-compatible version with bit masking:
kernel void functionGradient(const global float x1, const global float x2, const global float x3, const global float x4, const global float x5, global float* gradient) {
const float gid = get_global_id(0);
const float cosx1 = cos(x1);
const float cosx2 = cos((gid!=1)*x2+(gid==1)*3.1415927f);
const float expmx3 = exp(-x3);
gradient[gid] = (gid==0)*cosx1 + (gid<=1)*cosx2 + (gid==2)*(expmx3-x3*expmx3) + (gid>=3);
}
Note that there is no additional global/local memory access and all the (mutually exclusive) weighting factors are functions of the gloal ID. Each thread computes exactly the same thing (2 cos, 1 exp and a fes multiplications/additions) without any branching. Trigonometric functions / divisions take much more time than multiplications/additions, so as few as possible should be used by pre-calculating terms.