Some OpenMP related documents state that in order for a loop to be treated by OpenMP it must be “countable” providing different definitions for a loop being “countable”:
• the number of iterations in the loop must be countable
with an integer and loop use a fixed increment.
• the loop count can be “determined” ( what does it mean “determined”? )
Is it indeed the requirement of OpenMP? Or is it requirement of a specific compiler implementation of OpenMP?
Also, can the following loop - doesn't seems to be countable - be parallelized by OpenMP ( note that the question is if the code can be pararallelized and not if there is a way to create a parallel equivalent of this code )
for ( i = 0; i < cnt; )
{
x1 = 2.0 * x - 1.;
if ( x1 < 1.0 )
{
i = i + 3;
x = x*2.;
}
else // if ( x1 >= 1. )
{
i = i + 2;
x = x/2.;
}
}
The OpenMP standard requires the Canonical Loop Form
The canonical form allows the iteration count of all associated loops to be computed before executing the outermost loop. The computation is performed for each loop in an integer type.
This isn't exactly restricted to integers, you may use pointer types in C and random access iterator types in C++ which are logically translated to integers.
The loop variable must not be modified within the for loop (except for the increment expression). Therefore your example is not valid OpenMP code.
Further restrictions are that the loop test must be a relation comparison <, <=, >, or >= - not != or something more complex. The increment must be constant throughout the loop. The missing increment expression in your example also invalidates it.
You can read about this in the OpenMP standard section 2.6.
Related
What are "sequence points"?
What is the relation between undefined behaviour and sequence points?
I often use funny and convoluted expressions like a[++i] = i;, to make myself feel better. Why should I stop using them?
If you've read this, be sure to visit the follow-up question Undefined behavior and sequence points reloaded.
(Note: This is meant to be an entry to Stack Overflow's C++ FAQ. If you want to critique the idea of providing an FAQ in this form, then the posting on meta that started all this would be the place to do that. Answers to that question are monitored in the C++ chatroom, where the FAQ idea started out in the first place, so your answer is very likely to get read by those who came up with the idea.)
C++98 and C++03
This answer is for the older versions of the C++ standard. The C++11 and C++14 versions of the standard do not formally contain 'sequence points'; operations are 'sequenced before' or 'unsequenced' or 'indeterminately sequenced' instead. The net effect is essentially the same, but the terminology is different.
Disclaimer : Okay. This answer is a bit long. So have patience while reading it. If you already know these things, reading them again won't make you crazy.
Pre-requisites : An elementary knowledge of C++ Standard
What are Sequence Points?
The Standard says
At certain specified points in the execution sequence called sequence points, all side effects of previous evaluations
shall be complete and no side effects of subsequent evaluations shall have taken place. (§1.9/7)
Side effects? What are side effects?
Evaluation of an expression produces something and if in addition there is a change in the state of the execution environment it is said that the expression (its evaluation) has some side effect(s).
For example:
int x = y++; //where y is also an int
In addition to the initialization operation the value of y gets changed due to the side effect of ++ operator.
So far so good. Moving on to sequence points. An alternation definition of seq-points given by the comp.lang.c author Steve Summit:
Sequence point is a point in time at which the dust has settled and all side effects which have been seen so far are guaranteed to be complete.
What are the common sequence points listed in the C++ Standard?
Those are:
at the end of the evaluation of full expression (§1.9/16) (A full-expression is an expression that is not a subexpression of another expression.)1
Example :
int a = 5; // ; is a sequence point here
in the evaluation of each of the following expressions after the evaluation of the first expression (§1.9/18) 2
a && b (§5.14)
a || b (§5.15)
a ? b : c (§5.16)
a , b (§5.18) (here a , b is a comma operator; in func(a,a++) , is not a comma operator, it's merely a separator between the arguments a and a++. Thus the behaviour is undefined in that case (if a is considered to be a primitive type))
at a function call (whether or not the function is inline), after the evaluation of all function arguments (if any) which
takes place before execution of any expressions or statements in the function body (§1.9/17).
1 : Note : the evaluation of a full-expression can include the evaluation of subexpressions that are not lexically
part of the full-expression. For example, subexpressions involved in evaluating default argument expressions (8.3.6) are considered to be created in the expression that calls the function, not the expression that defines the default argument
2 : The operators indicated are the built-in operators, as described in clause 5. When one of these operators is overloaded (clause 13) in a valid context, thus designating a user-defined operator function, the expression designates a function invocation and the operands form an argument list, without an implied sequence point between them.
What is Undefined Behaviour?
The Standard defines Undefined Behaviour in Section §1.3.12 as
behavior, such as might arise upon use of an erroneous program construct or erroneous data, for which this International Standard imposes no requirements 3.
Undefined behavior may also be expected when this
International Standard omits the description of any explicit definition of behavior.
3 : permissible undefined behavior ranges from ignoring the situation completely with unpredictable results, to behaving during translation or program execution in a documented manner characteristic of the environment (with or with-
out the issuance of a diagnostic message), to terminating a translation or execution (with the issuance of a diagnostic message).
In short, undefined behaviour means anything can happen from daemons flying out of your nose to your girlfriend getting pregnant.
What is the relation between Undefined Behaviour and Sequence Points?
Before I get into that you must know the difference(s) between Undefined Behaviour, Unspecified Behaviour and Implementation Defined Behaviour.
You must also know that the order of evaluation of operands of individual operators and subexpressions of individual expressions, and the order in which side effects take place, is unspecified.
For example:
int x = 5, y = 6;
int z = x++ + y++; //it is unspecified whether x++ or y++ will be evaluated first.
Another example here.
Now the Standard in §5/4 says
Between the previous and next sequence point a scalar object shall have its stored value modified at most once by the evaluation of an expression.
What does it mean?
Informally it means that between two sequence points a variable must not be modified more than once.
In an expression statement, the next sequence point is usually at the terminating semicolon, and the previous sequence point is at the end of the previous statement. An expression may also contain intermediate sequence points.
From the above sentence the following expressions invoke Undefined Behaviour:
i++ * ++i; // UB, i is modified more than once btw two SPs
i = ++i; // UB, same as above
++i = 2; // UB, same as above
i = ++i + 1; // UB, same as above
++++++i; // UB, parsed as (++(++(++i)))
i = (i, ++i, ++i); // UB, there's no SP between `++i` (right most) and assignment to `i` (`i` is modified more than once btw two SPs)
But the following expressions are fine:
i = (i, ++i, 1) + 1; // well defined (AFAIK)
i = (++i, i++, i); // well defined
int j = i;
j = (++i, i++, j*i); // well defined
Furthermore, the prior value shall be accessed only to determine the value to be stored.
What does it mean? It means if an object is written to within a full expression, any and all accesses to it within the same expression must be directly involved in the computation of the value to be written.
For example in i = i + 1 all the access of i (in L.H.S and in R.H.S) are directly involved in computation of the value to be written. So it is fine.
This rule effectively constrains legal expressions to those in which the accesses demonstrably precede the modification.
Example 1:
std::printf("%d %d", i,++i); // invokes Undefined Behaviour because of Rule no 2
Example 2:
a[i] = i++ // or a[++i] = i or a[i++] = ++i etc
is disallowed because one of the accesses of i (the one in a[i]) has nothing to do with the value which ends up being stored in i (which happens over in i++), and so there's no good way to define--either for our understanding or the compiler's--whether the access should take place before or after the incremented value is stored. So the behaviour is undefined.
Example 3 :
int x = i + i++ ;// Similar to above
Follow up answer for C++11 here.
This is a follow up to my previous answer and contains C++11 related material..
Pre-requisites : An elementary knowledge of Relations (Mathematics).
Is it true that there are no Sequence Points in C++11?
Yes! This is very true.
Sequence Points have been replaced by Sequenced Before and Sequenced After (and Unsequenced and Indeterminately Sequenced) relations in C++11.
What exactly is this 'Sequenced before' thing?
Sequenced Before(§1.9/13) is a relation which is:
Asymmetric
Transitive
between evaluations executed by a single thread and induces a strict partial order1
Formally it means given any two evaluations(See below) A and B, if A is sequenced before B, then the execution of A shall precede the execution of B. If A is not sequenced before B and B is not sequenced before A, then A and B are unsequenced 2.
Evaluations A and B are indeterminately sequenced when either A is sequenced before B or B is sequenced before A, but it is unspecified which3.
[NOTES]
1 : A strict partial order is a binary relation "<" over a set P which is asymmetric, and transitive, i.e., for all a, b, and c in P, we have that:
........(i). if a < b then ¬ (b < a) (asymmetry);
........(ii). if a < b and b < c then a < c (transitivity).
2 : The execution of unsequenced evaluations can overlap.
3 : Indeterminately sequenced evaluations cannot overlap, but either could be executed first.
What is the meaning of the word 'evaluation' in context of C++11?
In C++11, evaluation of an expression (or a sub-expression) in general includes:
value computations (including determining the identity of an object for glvalue evaluation and fetching a value previously assigned to an object for prvalue evaluation) and
initiation of side effects.
Now (§1.9/14) says:
Every value computation and side effect associated with a full-expression is sequenced before every value computation and side effect associated with the next full-expression to be evaluated.
Trivial example:
int x;
x = 10;
++x;
Value computation and side effect associated with ++x is sequenced after the value computation and side effect of x = 10;
So there must be some relation between Undefined Behaviour and the above-mentioned things, right?
Yes! Right.
In (§1.9/15) it has been mentioned that
Except where noted, evaluations of operands of individual operators and of subexpressions of individual expressions are unsequenced4.
For example :
int main()
{
int num = 19 ;
num = (num << 3) + (num >> 3);
}
Evaluation of operands of + operator are unsequenced relative to each other.
Evaluation of operands of << and >> operators are unsequenced relative to each other.
4: In an expression that is evaluated more than once during the execution
of a program, unsequenced and indeterminately sequenced evaluations of its subexpressions need not be performed consistently in different evaluations.
(§1.9/15)
The value computations of the operands of an
operator are sequenced before the value computation of the result of the operator.
That means in x + y the value computation of x and y are sequenced before the value computation of (x + y).
More importantly
(§1.9/15) If a side effect on a scalar object is unsequenced relative to either
(a) another side effect on the same scalar object
or
(b) a value computation using the value of the same scalar object.
the behaviour is undefined.
Examples:
int i = 5, v[10] = { };
void f(int, int);
i = i++ * ++i; // Undefined Behaviour
i = ++i + i++; // Undefined Behaviour
i = ++i + ++i; // Undefined Behaviour
i = v[i++]; // Undefined Behaviour
i = v[++i]: // Well-defined Behavior
i = i++ + 1; // Undefined Behaviour
i = ++i + 1; // Well-defined Behaviour
++++i; // Well-defined Behaviour
f(i = -1, i = -1); // Undefined Behaviour (see below)
When calling a function (whether or not the function is inline), every value computation and side effect associated with any argument expression, or with the postfix expression designating the called function, is sequenced before execution of every expression or statement in the body of the called function. [Note: Value computations and side effects associated with different argument expressions are unsequenced. — end note]
Expressions (5), (7) and (8) do not invoke undefined behaviour. Check out the following answers for a more detailed explanation.
Multiple preincrement operations on a variable in C++0x
Unsequenced Value Computations
Final Note :
If you find any flaw in the post please leave a comment. Power-users (With rep >20000) please do not hesitate to edit the post for correcting typos and other mistakes.
C++17 (N4659) includes a proposal Refining Expression Evaluation Order for Idiomatic C++
which defines a stricter order of expression evaluation.
In particular, the following sentence
8.18 Assignment and compound assignment operators:....
In all cases, the assignment is sequenced after the value
computation of the right and left operands, and before the value computation of the assignment expression.
The right operand is sequenced before the left operand.
together with the following clarification
An expression X is said to be sequenced before an expression Y if every
value computation and every side effect associated with the expression X is sequenced before every value
computation and every side effect associated with the expression Y.
make several cases of previously undefined behavior valid, including the one in question:
a[++i] = i;
However several other similar cases still lead to undefined behavior.
In N4140:
i = i++ + 1; // the behavior is undefined
But in N4659
i = i++ + 1; // the value of i is incremented
i = i++ + i; // the behavior is undefined
Of course, using a C++17 compliant compiler does not necessarily mean that one should start writing such expressions.
I am guessing there is a fundamental reason for the change, it isn't merely cosmetic to make the old interpretation clearer: that reason is concurrency. Unspecified order of elaboration is merely selection of one of several possible serial orderings, this is quite different to before and after orderings, because if there is no specified ordering, concurrent evaluation is possible: not so with the old rules. For example in:
f (a,b)
previously either a then b, or, b then a. Now, a and b can be evaluated with instructions interleaved or even on different cores.
In C99(ISO/IEC 9899:TC3) which seems absent from this discussion thus far the following steteents are made regarding order of evaluaiton.
[...]the order of evaluation of subexpressions and the order in which
side effects take place are both unspecified. (Section 6.5 pp 67)
The order of evaluation of the operands is unspecified. If an attempt
is made to modify the result of an assignment operator or to access it
after the next sequence point, the behavior[sic] is undefined.(Section
6.5.16 pp 91)
I am writing a program in which I want to calculate set bits in an integer.
For eg, if 5 can be written as 0101 where a number of set bits is 2.
However, when I executed the program where I wrote count = count + n & 1 It didn't work but if I change that to count += n&1 it worked totally fine, and I couldn't find any reason for that
#include<bits/stdc++.h>
using namespace std;
int main(){
unsigned int n = 5;
unsigned int count = 0;
while(n){
//count += n&1;
count = count + n & 1;
n>>=1;
}
cout<<count;
return 0;
}
Precedence in C++ or any other programming languages determines the order of evaluation of those instructions, some go from right to left, some are evaluated first before the other. So, in this case, your += was evaluated after the & (AND) operator, and that was the reason for your wrong answer when is done this way
count = count + n & 1;
Because in the code above, the + is evaluated then the AND was then next. You can either put a bracket around it like this below
count = count + (n & 1);
Since the bracket operator has higher precedence it will always ensure you get the right results.
You can learn more about precedence here. You will see the order of evaluation for each operator.
The associativity of an operator is a property that determines how
operators of the same precedence are grouped in the absence of
parentheses. This affects how an expression is evaluated.
And you can always check your C++ or C reference Manuals to learn more about precedence.
https://en.cppreference.com/w/cpp/language/operator_precedence
Operators are evaluated from top to bottom of the list. Those at the
top are evaluated first, and those below are evaluated last according
to their associativity either left to right or right to left.
According to microsoft, the operators in C++ are the same in visual studio C++ 2010
http://msdn.microsoft.com/en-us/library/x04xhy0h.aspx
However, look at the following builds:
int^ number = 32;
if (number < 100)
MessageBox::Show("The number is not greater than 100");
Build failed
'<' : 'System::Int32 ^' does not define this operator or a conversion to a type acceptable to the predefined operator
if (number <= 100)
MessageBox::Show("The number is not greater than 100");
Build failed
'<=' : 'System::Int32 ^' does not define this operator or a conversion to a type acceptable to the predefined operator
if (number == 32)
MessageBox::Show("The is equal to 32");
Build successful... However the message is not displayed.
if (number = 32)
MessageBox::Show("The is equal to 32");
Build successful.. The message is displayed. (Why? The operator of the equality is ==)
Why is this happening?
int^ declares a handle to an object. Whenever you reference number directly, you're actually referencing a boxed integer (somewhat equivalent to (object)32 in C#).
In addition, handles to objects don't define the < or <= (or > or >=) operators when comparing to an integer literal. The reason for that can be deducted from the following:
They do, however define an == operator. But in order to compare, the literal value you're comparing to will be implicitly boxed, making the comparison (somewhat) equivalent to this C# code:
object number = 32;
if (number == (object)32)
MessageBox.Show("The number is equal to 32");
That comparison will check if the references are the same. Which they aren't - they're two different objects. Hence:
int^ number = 32;
if (number == 32)
MessageBox::Show("The number is equal to 32"); // isn't displayed
... and since you're comparing references rather than values, >, >=, <=, < would make little sense.
In your last case, you're assigning 32 to number, then checking if the result of that expression (which is itself 32) is different from 0 - it is, so the message is displayed. That's what if does in C++ (and C) - in C#, number = 32 does have the result 32, but you'd get a compiler error due to the if requiring a boolean value.
"Solution": Dereference the int^:
if (*number == 32)
MessageBox::Show("The number is equal to 32");
... or simply use int:
int number = 32;
EDIT: Rewrote based on Ben Voigt's more correct explanation.
On math.SE, a question about math notation enerated a discussion of how programming languages interpret the set {1,...,n} when n=0
The question asked for a mathematical notation to represent the R code 1:n
According to the comments, the mathematical interpretation of {1,...,n} when n=0 is that this is an empty set. A subsequent comment suggested that C is consistent with this interpretation, because for (int i = 1; i < n; i++) returns a empty set because it iterates 0 times.
It is not clear to me what the equivalent statement in R is, but 1:0 returns the vector [1,0]
Thus, for (i in 1:0) print(i) iterates over 1 and 0 (I interpret as analogous to the C code above)
Is this because {1,...,n} is not the correct notation for 1:n?
Does this mean R violates a universal rule?
Is there a consistent interpretation for this set among programming languages?
Each mathematical formalism has its own notation. To suggest that there is a "universal notation" is very "un-mathematical". Look at the notation associated with tensors or groups if you want examples of mathematical domains where multiple notational systems exist.
In R the code x <- 1:0 returns the ordered vector c(1,0). Just as the code x <- 2:-2 returns c(2,1,0,-1,-2). The code x <- seq(1, length=0) returns a sequence of length 0 which is printed in console sessions as integer(0). R is not really designed to mimic set notation but it does have some set functions and it also has packages that more fully implement set notation.
C has no concept of a set that a for loop runs over. A for loop for(a;b;c) d; is simply syntactic sugar for:
a;
loop: if (!b) goto done;
d;
c;
goto loop;
done: ;
See also my response at: Sequence construction that creates an empty sequence if lower is greater than upper bound - in R, seq_len(n) should be used in preference to 1:n for exactly this reason (the latter fails misbehaves when n=0).
some languages support the concept of ranges, in C it is arbitary what you make a for loop do, you could make it mean 0 or you could make it count backwards. In other languages a range that has the second number less that the first often produces a number sequence that is decreasing. But its arbitrary, and there is no universal rule.
I have a scenario like:
for (i = 0; i < n; i++)
{
for (j = 0; j < m; j++)
{
for (k = 0; k < x; k++)
{
val = 2*i + j + 4*k
if (val != 0)
{
for(t = 0; t < l; t++)
{
someFunction((i + t) + someFunction(j + t) + k*t)
}
}
}
}
}
Considering this is block A, Now I have two more similar blocks in my code. I want to put them in parallel, so I used OpenMP pragmas. However I am not able to parallelize it, because I am a tad confused that which variables would be shared and private in this case. If the function call in the inner loop was an operation like sum += x, then I could have added a reduction clause.
In general, how would one approach parallelizing a code using OpenMP, when we there is a nested for loop, and then another inner for loop doing the main operation.
I tried declaring a parallel region, and then simply putting pragma fors before the blocks, but definitely I am missing a point there!
Thanks,
Sayan
I'm more of a Fortran programmer than C so my knowledge of OpenMP in C-style is poor, and I'll leave the syntax to you.
Your easiest approach here is probably (I'll qualify this later) to simply parallelise the outermost loop. By default OpenMP will regard variable i as private, all the rest as shared. This is probably not what you want, you probably want to make j and k and t private too. I suspect that you want val private also.
I'm a bit puzzled by the statement at the bottom of your nest of loops (ie someFunction...), which doesn't seem to return any value at all. Does it work by side-effects ?
So, you shouldn't need to declare a parallel region enclosing all this code, and you should probably only parallelise the outermost loop. If you were to parallelise the inner loops too you might find your OpenMP installation either ignoring them, spawning more processes than you have processors, or complaining bitterly.
I say that your easiest approach is probably to parallelise the outermost loop because I've made some assumptions about what your program (fragment) is doing. If the assumptions are wrong you might want to parallelise one of the inner loops. Another point to check is that the number of executions of the loop(s) you parallelise is much greater than the number of threads you use. You don't want to have OpenMP run loops with a trip count of, say, 7, on 4 threads, the load balance would be very poor.
You're correct, the innermost statement would rather be someFunction((i + t) + someFunction2(j + t) + k*t).