What is the running time of this algorithm:
for i=1 to n^2
for j=1 to i
// some constant time operation
I want to say O(n^4) but I can't be certain. How do you figure this out?
n^4 is correct. The inner loop takes an average of (n^2)/2 time to run, because i goes up to n^2 linearly, and it is run (n^2) times.
You are correct, it is N^4.
Do the substitution M = N^2. Now your loops change to this:
for i in 0..M
for j in 0..i
This is your familiar O(M^2), hence the result is O((N^2)^2) = O(N^4) after the reverse substitution.
The constant time operation is run:
1 + 2 + 3 + ... + n^2 (n^2 adders)
times which is less than:
n^2 + n^2 + ... + n^2 (n^2 adders)
= n^2 * n^2
= n^4
So, it's obviously O(n^4)
To prove it's Θ(n^4), you can use a liitle math:
1 + 2 + 3 + ... + n^2
= n^2 * (n^2 + 1) / 2
= n^4 / 2 + n^2 / 2
>= n^4 / 2
With nested loops the Big Oh run time multiplicative. So Big Oh of the outer loop (N^2) is multiplied by the Big Oh of the inner (N^2). Therefore the Big Oh is (N^2 * N^2) and if you remember how to add exponents of a similar base you get N^(2+2) or N^4.
Using Sigma Notation, you end up getting the order of growth methodically:
n^5 = outer * inner
outer = n^2
inner = n^2 + n^2-1 + n^2-2 +...1
Related
Somehow, I find that it is much harder to derive Big O complexities for recursive algorithms compared to iterative algorithms. Do provide some insight about how I should go about solving these 2 questions.
*assume that submethod has linear complexity
def myMethod(n)
if (n>0)
submethod(n)
myMethod(n/2)
end
end
def myMethod(k,n)
if(n>0)
submethod(k)
myMethod(k,n/2)
end
end
For your first problem, the recurrence will be:
T(n) = n + T(n/2)
T(n/2) = n/2 + T(n/4)
...
...
...
T(2) = 2 + T(1)
T(1) = 1 + T(0) // assuming 1/2 equals 0(integer division)
adding up we get:
T(n) = n + n/2 + n/4 + n/8 + ..... 1 + T(0)
= n(1 + 1/2 + 1/4 + 1/8 .....) + k // assuming k = T(0)
= n*1/(1 - 1/2) ( sum of geometric series a/(1-r) when n tends to infinity)
= 2n + k
Therefore, T(n) = O(n). Remember i have assumed n tends to infinity ,cause this is what we do in Asymptotic analysis.
For your second problem its easy to see that, we perform k primitive operations everytime till n becomes 0. This happens log(n) times. Therefore, T(n) = O(k*log(n))
All you need to do is count how many times a basic operation is executed. This is true for analysing any kind of algorithm. In your case, we will count the number of times submethod is called.
You could break-down the running time of call myMethod(n) to be 1 + myMethod(n / 2). Which you can further break down to 1 + (1 + myMethod(n / 4)). At some point you will reach the base case, in log(n)th step. That gives you an algorithm of log(n).
The second one is no different, since k is constant all the time, it will again take log(n) time, assuming submethod takes constant time regardless of its input.
I am trying to count the cost of the following algorithm in terms of a function of n.
for i:= 1 to n do
for j:= i to n do
k:=0
I understand that the inner for loop will iterate (n-1) + (n-2) + .... (n-n) times, however I don't know how to express this mathematically in a simpler form. How can I do this ?
(n-1) + (n-2) + .... (n-n) is equal to the sum of all integers from 0 to N-1. So it is equal to the N-1th triangular number, which can be found with the formula
Tn = n * (n+1) / 2
Which is equivalent to (1/2)*n^2 + (1/2)*n.
When calculating Big O complexity, you discard constant multipliers and all but the fastest-growing component, so an algorithm that takes (1/2)*n^2 + (1/2)*n steps to execute runs in O(n^2) time.
The inner loop, on average iterates (≈½n) times.
In "Big O" notation, you only care about the largest factor.
That is, for example, if you have:
n³ + n + log(n) + 1234
then the only thing that matters is the n³ factor, so O(n³).
So in your case:
½n x n = ½n²
which is O(n²) because the ½ doesn't matter.
What is the time complexity for the following function?
for(int i = 0; i < a.size; i++) {
for(int j = i; j < a.size; i++) {
//
}
}
I think it is less than big O n^2 because we arent iterating over all of the elements in the second for loop. I believe the time complexity comes out to be something like this:
n[ (n) + (n-1) + (n-2) + ... + (n-n) ]
But when I solve this formula it comes out to be
n^2 - n + n^2 - 2n + n^2 - 3n + ... + n^2 - n^2
Which doesn't seem correct at all. Can somebody tell me exactly how to solve this problem, and where I am wrong.
That is O(n^2). If you consider the iteration where i = a.size() - 1, and you work your way backwards (i = a.size() - 2, i = a.size - 3, etc), you are looking at the following sum of number of iterations, where n = a.size.
1 + 2 + 3 + 4 + ... + n
The sum of this series is n(n+1)/2, which is O(n^2). Note that big-O notation ignores constants and takes the highest polynomial power when it is applied to a polynomial function.
It will run for:
1 + 2 + 3 + .. + n
Which is 1/2 n(n+1) which give us O(n^2)
The Big-O notation will only keep the dominant term, neglecting constants too
The Big-O is only used to compare algorithms on the same variation of a problem using the same complexity analysis standard, if and only if the dominant terms are different.
If the dominant terms are the same, you need to compare Big-Theta or Time complexity, which will show minor differences.
Example
A
for i = 1 .. n
for j = i .. n
..
B
for i = 1 .. n
for j = 1 .. n
..
We have
Time(A) = 1/2 n(n+1) ~ O(n^2)
Time(B) = n^2 ~ O(n^2)
O(A) = O(B)
T(A) < T(B)
Analysis
To visualize how we got 1 + 2 + 3 + .. n:
for i = 1 .. n:
print "(1 + "
sum = 0
for j = i .. n:
sum++
print sum") + "
will print the following:
(1+n) + (1+(n-1)) + .. + (1+3) + (1+2) + (1+1) + (1+0)
n+1 + n + n-1 + .. + 3 + 2 + 1
1 + 2 + 3 + .. + n + n+1
1/2 n(n+1) + (n+1)
1/2 n^2 + 1/2 n + n + 1
1/2 n^2 + 3/2 n + 1
Yes, the number of iterations is strictly less than n^2, but it's still Θ(n^2). It will eventually be greater than n^k for any k<2, and it will eventually be less than n^k for any k>2.
(As a side note, computer scientists often say big-O when they really mean big-theta (Θ). It's technically correct to say that almost every algorithm you've seen has O(n!) running time; all reasonably algorithms have running times that grow no more quickly than n!. But it's not really useful to say that the complexity is O(n!) if it's also O(n log n), so by some kind of Gricean maxim we assume that when someone says an algorithm's complexiy is O(f(x)) that f(x) is as small as possible.)
I'm trying to learn how to find the big-theta bounds of various algorithms, but I'm having a hard time understanding how to do it, even after reading a number of questions here and lectures and textbooks on the subject. So for example
procedure for array a{
step=1
while(step <= n){
i = n
while(i >= step){
a[i]= a[i-step] + a[i]
i = i - 1}
step = step * 2}
}
I want to figure out the big-theta bound on the number of additions this goes through in terms of n, the number of indices in array a. I can see that the outer loop itself goes through log(n) iterations, but I can't figure out how to express what happens in the inner loop. Does anyone have an explanation or perhaps even a resource I might try consulting?
Big Theta notation asks us to find 2 constants, k1 and k2 such that our function f(n) is between k1*g(n) and k2*g(n) for sufficiently large n. In other words, can we find some other function g(n) that is at some point less than f(n) and at another point greater than f(n) (monotonically each way).
To prove Big-Theta, we need to find g(n) such that f(n) is O(g(n)) (upper bound), and f(n) is Big-Omega(g(n)) (lower bound).
Prove Big-O
In terms of Big-O notation (where f(n) <= g(n)*k), your algorithm, f(n), is O(log(n)*n), In this case g(n) = log(n) * n.
Let's prove this:
Find Inner Loop Executions
How many times does the outer loop execute? Track the "step" variable:
Let's say that n is 100:
1
2
4
8
16
32
64
128 (do not execute this loop)
That's 7 executions for an input of 100. We can equivalently say that it executes (log n) times (actually floor(log n) times, but log(n) is adequate).
Now let's look at the inner loop. Track the i variable, which starts at n and decrements until it is of size step each iteration. Therefore, the inner while loop will execute n - step times, for each value of step.
For example, when n = 100
100 - 1 = 99 iterations
100 - 2 = 98 iterations
100 - 4 = 96 iterations
100 - 8 = 92 iterations
100 - 16 = 84 iterations
100 - 32 = 68 iterations
100 - 64 = 36 iterations
So what's our total iteration count of the inner loop?
(n-1)
(n-1) + (n-2)
(n-1) + (n-2) + (n-4)
(n-1) + (n-2) + (n-4) + (n-8)
etc.
How is this thing growing? Well, because we know the outer loop will iterate log(n) times, we can formulate this thing as a summation:
Sum(from i=0 to log(n)) n-2^i
= log(n)*n - Sum(from i=0 to log(n)) 2^i
= log(n)*n - (2^0 + 2^1 + 2^2 + ... + 2^log(n))
= log(n)*n - ( (1-2^log(n) ) / (1-2) ) (actually 2^log(n+1) but close enough)
= log(n)*n + 1 - n
So now our goal is to show that:
log(n)*n + 1 - n = O(log(n)*n)
Clearly, log(n)*n is O(log(n)*n), but what about the 1-n?
1-n = O(log(n)*n)?
1-n <= k*log(n)*n, for some k?
Let k = 1
1-n <= log(n)*n?
Add n to both sides
1 <= n*log(n) + n? YES
So we've shown that f(n) is O(n*log(n)).
Prove Big-Omega
Now that we have an upper bound on f(n) using log(n) * n, let's try to get a lower bound on f(n) also using log(n) * n.
For a lower bound we use Big Omega notation. Big Omega looks for a function g(n)*k <= f(n) for some positive constant k.
k(n*log(n)) <= n*log(n) + 1 - n?
let k = 1/10
n*log(n)/10 <= n*log(n) + 1 - n?
(n*log(n) - 10n*log(n)) / 10 <= 1 - n?
-9n*log(n)/10 <= 1 - n? Multiply through by 10
-9n*log(n) <= 10 - 10n? Multiply through by -1 (flipping inequality)
9n*log(n) >= 10n - 10? Divide through by 9
n*log(n) >= 10n/9 - 10/9? Divide through by n
log(n) >= 10/9 - 10/9n ? Yes
Clearly, the quantity log(n) grows larger as (10/9 - 10/9n) tends towards 10/9. In fact for n = 1, 0 >= 10/9 - 10/9. 0 >= 0.
Prove Big-Theta
So now we've shown that f(n) is Big-Omega(n*log(n)). Combining this with the proof for f(n) is O(n*log(n)), and we've shown that f(n) is Big-Theta(n*log(n))! (the exclamation point is for excitement, not factorial...)
g(n) = n*log(n), and one valid set of constants is k1=1/10 (lower bound) and k2 = 1 (upper bound).
To prove big-O: there are floor(log2(n)) + 1 = O(log(n)) iterations through the outer loop, and the inner loop iterates O(n) times per, for a total of O(n * log(n)).
To prove big-Omega: there are floor(log2(n/2)) + 1 = Omega(log(n)) iterations through the outer loop during which step <= n/2. The inner loop iterates n + 1 - step times, which, for these outer iterations, is more than n/2 = Omega(n) per, for a total of Omega(n * log(n)).
Together, big-O and big-Omega prove big-Theta.
Simplifying the representation of your code like the following, we can translate your code into Sigma notation
for (step = 1; <= n; step = step * 2) {
for(i = n; i >= step; step = step - 1) {
}
}
Like this:
I am to show that log(n!) = Θ(n·log(n)).
A hint was given that I should show the upper bound with nn and show the lower bound with (n/2)(n/2). This does not seem all that intuitive to me. Why would that be the case? I can definitely see how to convert nn to n·log(n) (i.e. log both sides of an equation), but that's kind of working backwards.
What would be the correct approach to tackle this problem? Should I draw the recursion tree? There is nothing recursive about this, so that doesn't seem like a likely approach..
Remember that
log(n!) = log(1) + log(2) + ... + log(n-1) + log(n)
You can get the upper bound by
log(1) + log(2) + ... + log(n) <= log(n) + log(n) + ... + log(n)
= n*log(n)
And you can get the lower bound by doing a similar thing after throwing away the first half of the sum:
log(1) + ... + log(n/2) + ... + log(n) >= log(n/2) + ... + log(n)
= log(n/2) + log(n/2+1) + ... + log(n-1) + log(n)
>= log(n/2) + ... + log(n/2)
= n/2 * log(n/2)
I realize this is a very old question with an accepted answer, but none of these answers actually use the approach suggested by the hint.
It is a pretty simple argument:
n! (= 1*2*3*...*n) is a product of n numbers each less than or equal to n. Therefore it is less than the product of n numbers all equal to n; i.e., n^n.
Half of the numbers -- i.e. n/2 of them -- in the n! product are greater than or equal to n/2. Therefore their product is greater than the product of n/2 numbers all equal to n/2; i.e. (n/2)^(n/2).
Take logs throughout to establish the result.
Sorry, I don't know how to use LaTeX syntax on stackoverflow..
See Stirling's Approximation:
ln(n!) = n*ln(n) - n + O(ln(n))
where the last 2 terms are less significant than the first one.
For lower bound,
lg(n!) = lg(n)+lg(n-1)+...+lg(n/2)+...+lg2+lg1
>= lg(n/2)+lg(n/2)+...+lg(n/2)+ ((n-1)/2) lg 2 (leave last term lg1(=0); replace first n/2 terms as lg(n/2); replace last (n-1)/2 terms as lg2 which will make cancellation easier later)
= n/2 lg(n/2) + (n/2) lg 2 - 1/2 lg 2
= n/2 lg n - (n/2)(lg 2) + n/2 - 1/2
= n/2 lg n - 1/2
lg(n!) >= (1/2) (n lg n - 1)
Combining both bounds :
1/2 (n lg n - 1) <= lg(n!) <= n lg n
By choosing lower bound constant greater than (1/2) we can compensate for -1 inside the bracket.
Thus lg(n!) = Theta(n lg n)
Helping you further, where Mick Sharpe left you:
It's deriveration is quite simple:
see http://en.wikipedia.org/wiki/Logarithm -> Group Theory
log(n!) = log(n * (n-1) * (n-2) * ... * 2 * 1) = log(n) + log(n-1) + ... + log(2) + log(1)
Think of n as infinitly big. What is infinite minus one? or minus two? etc.
log(inf) + log(inf) + log(inf) + ... = inf * log(inf)
And then think of inf as n.
Thanks, I found your answers convincing but in my case, I must use the Θ properties:
log(n!) = Θ(n·log n) => log(n!) = O(n log n) and log(n!) = Ω(n log n)
to verify the problem I found this web, where you have all the process explained: http://www.mcs.sdsmt.edu/ecorwin/cs372/handouts/theta_n_factorial.htm
http://en.wikipedia.org/wiki/Stirling%27s_approximation
Stirling approximation might help you. It is really helpful in dealing with problems on factorials related to huge numbers of the order of 10^10 and above.
This might help:
eln(x) = x
and
(lm)n = lm*n
If you reframe the problem, you can solve this with calculus! This method was originally shown to me via Arthur Breitman https://twitter.com/ArthurB/status/1436023017725964290.
First, you take the integral of log(x) from 1 to n it is n*log(n) -n +1. This proves a tight upper bound since log is monotonic and for every point n, the integral from n to n+1 of log(n) > log(n) * 1. You can similarly craft the lower bound using log(x-1), as for every point n, 1*log(n) > the integral from x=n-1 to n of log(x). The integral of log(x) from 0 to n-1 is (n-1)*(log(n-1) -1), or n log(n-1) -n -log(n-1)+1.
These are very tight bounds!