I am trying to solve the following question on computational complexity:
Compute the computational complexity of the following algorithm and
write down its complexity in Big O, Big Omega and Theta
for i=1 to m {
x(i) =0;
for j=1 to n {
x(i) = x(i) + A(i,j) * b(j)
}
}
where A is mxn and b is nx1.
I ended up with Big O O(mn^2)
Big Omega(1) and Theta(mn^2).
Assuming that the following statement runs in constant time:
x(i) = x(i) + A(i,j) * b(j)
this is thus done in O(1), and does not depend on the values for i and j. Since you iterate over this statement in the inner for loop, exactly n times, you can say that the following code runs in O(n):
x(i) =0;
for j=1 to n {
meth1
}
(assuming the assignment is done in constant time as well). Again it does not depend on the exact value for i. Finally we take the outer loop into account:
for i=1 to m {
meth2
}
The method meth2 is repeated exactly m times, thus a tight upper bound for the time complexity in O(n m).
Since there are no conditional statements, nor recursive and the structure of the data A, b and x does not change the execution of the program, the algorithm is also big Omega(m n) and big Theta(m n).
Of course you can over-estimate big oh and under-estimate big omega: for every algorithm you can say it is Ω(1) and for some that it is O(2n), but the point is that you do not buy much with that.
Recall that f = Theta(g) if and only if f=O(g) and f=Omega(g).
The matrix-vector product can be computed in Theta(mn) time (assuming naive implementation) and the sum of vectors in O(m), so the total running time is Theta(mn). From here it follows that the time is also O(mn) and Omega(mn).
Related
I've got the following example:
i=2;
while i<=n {
O(1)
j=2*i
while j<=n {
O(1)
j=j+i
}
i=i+1
I'm a beginner at calculating asymptotic complexity. I think it is O((n-1)*(n/4)) but I'm not sure if this answer is correct or I'm missing something.
In the inner loop, j goes from 2i to n in steps i, so the inner loop runs (n-2i)/i+1 times, which is n/i-1 (integer division).
Then the outer loop runs with i from 2 to n, for a total of n/2-1+n/3-1+n/4-1+...n/(n/2)-1 inner iterations (for larger i, there are no iterations).
This quantity is difficult to estimate, but is bounded above by n (H(n/2)-1)-n/2 where H denotes an Harmonic Number.
We know that Hn~O(log n), hence, asymptotically, the running time is O(n log n).
I'm studying for an exam, and i've come across the following question:
Provide a precise (Θ notation) bound for the running time as a
function of n for the following function
for i = 1 to n {
j = i
while j < n {
j = j + 4
}
}
I believe the answer would be O(n^2), although I'm certainly an amateur at the subject but m reasoning is the initial loop takes O(n) and the inner loop takes O(n/4) resulting in O(n^2/4). as O(n^2) is dominating it simplifies to O(n^2).
Any clarification would be appreciated.
If you proceed using Sigma notation, and obtain T(n) equals something, then you get Big Theta.
If T(n) is less or equal, then it's Big O.
If T(n) is greater or equal, then it's Big Omega.
I am reading an example where the following are equivalent to O(N):
O(N + P), where P < N/2
O(N + log N)
Can someone explain in laymen terms how it is that the two examples above are the same thing as O(N)?
We always take the greater one in case of addition.
In both the cases N is bigger than the other part.
In first case P < N/2 < N
In second case log N < N
Hence the complexity is O(N) in both the cases.
Let f and g be two functions defined on some subset of the real numbers. One writes
f(x) = O(g(x)) as x -> infinite
if and only if there is a positive constant M such that for all sufficiently large values of x, the absolute value of f(x) is at most M multiplied by the absolute value of g(x). That is, f(x) = O(g(x)) if and only if there exists a positive real number M and a real number x0 such that
|f(x)| <= M |g(x)| for all x > x0
So in your case 1:
f(N) = N + P <= N + N/2
We could set M = 2 Then:
|f(N)| <= 3/2|N| <= 2|N| (N0 could any number)
So:
N+p = O(N)
In your second case, we could also set M=2 and N0=1 to satify that:
|N + logN| <= 2 |N| for N > 1
Big O notation usually only provides an upper bound on the growth rate of the function, wiki. Meaning for your both cases, as P < N and logN < N. So that O(N + P) = O(2N) = O(N), The same to O(N + log N) = O(2N) = O(N). Hope that can answer your question.
For the sake of understanding you can assume that O(n) represents that the complexity is of the order of n and also that O notation represents the upper bound(or the complexity in worst case). So, when I say that O(n+p) it represents that the order of n+p.
Let's assume that in worst case p = n/2, then what would be order of n+n/2? It would still be O(n), that is, linear because constants do form a part of the Big-O notation.
Similary, for O(n+logn) because logn can never be greater than n. So, overall complexity turns out to be linear.
In short
If N is a function and C is a constant:
O(N+N/2):
If C=2, then for any N>1 :
(C=2)*N > N+N/2,
2*N>3*N/2,
2> 3/2 (true)
O(N+logN):
If C=2, then for any N>2 :
(C=2)*N > N+logN,
2*N > N+logN,
2>(N+logN)/N,
2> 1 + logN/N (limit logN/N is 0),
2>1+0 (true)
Counterexample O(N^2):
No C exists such that C*N > N^2 :
C > N^2/N,
C>N (contradiction).
Boring mathematical part
I think the source of confusion is that equals sign in O(f(x))=O(N) does not mean equality! Usually if x=y then y=x. However consider O(x)=O(x^2) which is true, but reverse is false: O(x^2) != O(x)!
O(f(x)) is an upper bound of how fast a function is growing.
Upper bound is not an exact value.
If g(x)=x is an upper bound for some function f(x), then function 2*g(x) (and in general anything growing faster than g(x)) is also an upper bound for f(x).
The formal definition is: for function f(x) to be bound by some other function g(x) if you chose any constant C then, starting from some x_0 g(x) is always greater than f(x).
f(x)=N+N/2 is the same as 3*N/2=1.5*N. If we take g(x)=N and our constant C=2 then 2*g(x)=2*N is growing faster than 1.5*N:
If C=2 and x_0=1 then for any n>(x_0=1) 2*N > 1.5*N.
same applies to N+log(N):
C*N>N+log(N)
C>(N+logN)/N
C>1+log(N)/N
...take n_0=2
C>1+1/2
C>3/2=1.5
use C=2: 2*N>N+log(N) for any N>(n_0=2),
e.g.
2*3>3+log(3), 6>3+1.58=4.68
...
2*100>100+log(100), 200>100+6.64
...
Now interesting part is: no such constant exist for N & N^2. E.g. N squared grows faster than N:
C*N > N^2
C > N^2/N
C > N
obviously no single constant exists which is greater than a variable. Imagine such a constant exists C=C_0. Then starting from N=C_0+1 function N is greater than constant C, therefore such constant does not exist.
Why is this useful in computer science?
In most cases calculating exact algorithm time or space does not make sense as it would depend on hardware speed, language overhead, algorithm implementation details and many other factors.
Big O notation provides means to estimate which algorithm is better independently from real world complications. It's easy to see that O(N) is better than O(N^2) starting from some n_0 no matter which constants are there in front of two functions.
Another benefit is ability to estimate algorithm complexity by just glancing at program and using Big O properties:
for x in range(N):
sub-calc with O(C)
has complexity of O(N) and
for x in range(N):
sub-calc with O(C_0)
sub-calc with O(C_1)
still has complexity of O(N) because of "multiplication by constant rule".
for x in range(N):
sub-calc with O(N)
has complexity of O(N*N)=O(N^2) by "product rule".
for x in range(N):
sub-calc with O(C_0)
for y in range(N):
sub-calc with O(C_1)
has complexity of O(N+N)=O(2*N)=O(N) by "definition (just take C=2*C_original)".
for x in range(N):
sub-calc with O(C)
for x in range(N):
sub-calc with O(N)
has complexity of O(N^2) because "the fastest growing term determines O(f(x)) if f(x) is a sum of other functions" (see explanation in the mathematical section).
Final words
There is much more to Big-O than I can write here! For example in some real world applications and algorithms beneficial n_0 might be so big that an algorithm with worse complexity works faster on real data.
CPU cache might introduce unexpected hidden factor into otherwise asymptotically good algorithm.
Etc...
Say I have following algorithm:
for(int i = 1; i < N; i *= 3) {
sum++
}
I need to calculate the complexity using tilde-notation, which basically means that I have to find a tilde-function so that when I divide the complexity of the algorithm by this tilde-function, the limit in infinity has to be 1.
I don't think there's any need to calculate the exact complexity, we can ignore the constants and then we have a tilde-complexity.
By looking at the growth off the index, I assume that this algorithm is
~ log N
But rather than having a binary logarithmic function, the base in this case is 3.
Does this matter for the exact notation? Is the order of growth exactly the same and thus can we ignore the base when using Tilde-notation? Do I approach this correctly?
You are right, the for loop executes ceil(log_3 N) times, where log_3 N denotes the base-3 logarithm of N.
No, you cannot ignore the base when using the tilde notation.
Here's how we can derive the time complexity.
We will assume that each iteration of the for loop costs C, for some constant C>0.
Let T(N) denote the number of executions of the for-loop. Since at j-th iteration the value of i is 3^j, it follows that the number of iterations that we make is the smallest j for which 3^j >= N. Taking base-3 logarithms of both sides we get j >= log_3 N. Because j is an integer, j = ceil(log_3 N). Thus T(N) ~ ceil(log_3 N).
Let S(N) denote the time complexity of the for-loop. The "total" time complexity is thus C * T(N), because the cost of each of T(N) iterations is C, which in tilde notation we can write as S(N) ~ C * ceil*(log_3 N).
I have a simple algorithm that prints the two dimensional matrix (m*n, m and n are different numbers):
for(i=0;i<m;i++)
for(j=0;j<n;j++)
Console.WriteLine("{0}",A[i,j]);
I read that the big O notation for this algorithm is O(n^2);
Could somebody explain me what is the "n^2" in that statement? If this is number of elementary operations, then it should be m*n, not n^2?
In reality it should me m*n. We can assume it to be the number of elementary operations in this case, but the actual definition is its "the upper bound of the number of elementary operations."
Yeah, time complexity of for the specified block of code is O(n * m).
In simple words, that means your algo does <= k * n * m operations, k is some small constant factor.
With for-loops, the complexity is measured as O(N) * the block inside the for-loop. The first for-loop contains a second for-loop so the complexity would be 0(N) * 0(N) = O(N^2). The inner for-loop contains a simple output statement, which has a complexity of 0(1) The N corresponds to the number of inputs so the time taken to execute the code is proportional to the number of items squared.
first for loop runs for m-1 times and second for loop runs for n-1 times..
m-1 times = 1+2+3....m-1 times (same for second forloop)
we know that sum of natural numbers is x(x-1)/2 = x^2/2 - x/2
there are 2 loops so adding them gives you 2(x^2/2-x/2)
in Bing O notation we consider only the most dominant value and ignore coefficients, so we get x^2
so O(N) = x^2