Minimising number of calls to std::max in nested loop - parallel-processing

Question

I'm trying to reduce the number of calls to std::max in my inner loop, as I'm calling it millions of times (no exaggeration!) and that's making my parallel code run slower than the sequential code. The basic idea (yes, this IS for an assignment) is that the code calculates the temperature at a certain gridpoint, iteration by iteration, until the maximum change is no more than a certain, very tiny number (e.g 0.01). The new temp is the average of the temps in the cells directly above, below and beside it. Each cell has a different value as a result, and I want to return the largest change in any cell for a given chunk of the grid.
I've got the code working but it's slow because I'm doing a large (excessively so) number of calls to std::max in the inner loop and it's O(n*n). I have used a 1D domain decomposition
Notes: tdiff doesn't depend on anything but what's in the matrix
the inputs of the reduction function are the result of the lambda function
diff is the greatest change in a single cell in that chunk of the grid over 1 iteration
blocked range is defined earlier in the code
t_new is new temperature for that grid point, t_old is the old one
max_diff = parallel_reduce(range, 0.0,
//lambda function returns local max
[&](blocked_range<size_t> range, double diff)-> double
{
for (size_t j = range.begin(); j<range.end(); j++)
{
for (size_t i = 1; i < n_x-1; i++)
{
t_new[j*n_x+i]=0.25*(t_old[j*n_x+i+1]+t_old[j*n_x+i-1]+t_old[(j+1)*n_x+i]+t_old[(j-1)*n_x+i]);
tdiff = fabs(t_old[j*n_x+i] - t_new[j*n_x+i]);
diff = std::max(diff, tdiff);
}
}
return diff; //return biggest value of tdiff for that iteration - once per 'i'
},
//reduction function - takes in all the max diffs for each iteration, picks the largest
[&](double a, double b)-> double
{
convergence = std::max(a,b);
return convergence;
}
);
How can I make my code more efficient? I want to make less calls to std::max but need to maintain the correct values. Using gprof I get:
Each sample counts as 0.01 seconds.
% cumulative self self total
time seconds seconds calls ms/call ms/call name
61.66 3.47 3.47 3330884 0.00 0.00 double const& std::max<double>(double const&, double const&)
38.03 5.61 2.14 5839 0.37 0.96 _ZZ4mainENKUlN3tbb13blocked_rangeImEEdE_clES1_d
ETA: 61.66% of the time spent executing my code is on the std::max calls, it calls over 3 million times. The reduce function is called for every output of the lambda function, so reducing the number of calls to std::max in the lambda function will also reduce the number of calls to the reduce function

Answer 1

First of all, I would expect std::max to be inlined into its caller, so it's suspicious that gprof points it out as a separate hotspot. Do you maybe analyze a debug configuration?
Also, I do not think that std::max is a culprit here. Unless some special checks are enabled in its implementation, I believe it should be equivalent to (diff<tdiff)?tdiff:diff. Since one of the arguments to std::max is the variable that you update, you can try if (tdiff>diff) diff = tdiff; instead, but I doubt it will give you much (and perhaps compilers can do such optimization on their own).
Most likely, std::max is highlighted as the result of sampling skid; i.e. the real hotspot is in computations above std::max, which makes perfect sense, due to both more work and accesses to non-local data (arrays) that might have longer latency, especially if the corresponding locations are not in CPU cache.
Depending on the size of the rows (n_x) in your grid, processing it by rows like you do can be inefficient, cache-wise. It's better to reuse data from t_old as much as possible while those are in cache. Processing by rows, you either don't re-use a point from t_old at all until the next row (for i+1 and i-1 points) or only reuse it once (for two neighbors in the same row). A better approach is to process the grid by rectangular blocks, which helps to re-use data that are hot in cache. With TBB, the way to do that is to use blocked_range2d. It will need minimal changes in your code; basically, changing the range type and two loops inside the lambda: the outer and inner loops should iterate over range.rows() and range.cols(), respectively.

Answer 2

I ended up using parallel_for:
parallel_for(range, [&](blocked_range<size_t> range)
{
double loc_max = 0.0;
double tdiff;
for (size_t j = range.begin(); j<range.end(); j++)
{
for (size_t i = 1; i < n_x-1; i++)
{
t_new[j*n_x+i]=0.25*(t_old[j*n_x+i+1]+t_old[j*n_x+i-1]+t_old[(j+1)*n_x+i]+t_old[(j-1)*n_x+i]);
tdiff = fabs(t_old[j*n_x+i] - t_new[j*n_x+i]);
loc_max = std::max(loc_max, tdiff);
}
}
//reduction function - takes in all the max diffs for each iteration, picks the largest
{
max_diff = std::max(max_diff, loc_max);
}
}
);
And now my code runs in under 2 seconds for an 8000x8000 grid :-)