First of all sorry for asking such a basic question.
But I am having difficulties understanding substitution method for solving recurrences.I am following Introduction to Algo.s -CLRS. As I am not able to find enough examples and ambiguity is the main concern.Especially the induction step.In the text books we have to prove f(n) implies f(n+1) but in CLRS this step is missing or may be I am not getting the example. Please explain step by step how to prove that O(n^2) is the solution for recurrence function T(n)=T(n-1)+n
Its the general steps of substitution method that I want to understand. If you could shed some light on strong mathematical induction and provide links to material on substitution method that'll be helpful also.
In substitution method, simply replace any occurance of T(k) by T(k-1) + k, and do it iteratively.
T(n) = T(n-1) + n =
= (T(n-2) + (n-1)) + n =
= T(n-3) + (n-2) + (n-1) + n =
= ... =
= 1 + 2 + ... + n-1 + n
From sum of arithmetic progression, you can get that T(n) is in O(n^2).
Note that substitution method is usually used to get an intuition on what the complexity is, to formally prove it - you will probably need a different tool - such as mathematical induction.
The formal proof will go something like that:
Claim: T(n) <= n^2
Base: T(1) = 1 <= 1^2
Hypothesis: the claim is true for each `k < n` for some `n`.
T(n) = T(n-1) + n <= (hypothesis) (n-1)^2 + n = n^2-2n + 1 < n^2 (for n > 1)
Related
I don't think I have fully understand recurrence in algorithm.
Well, the n in the recurrence function can also be changed into n^2 or n^3. Are they just the same with the n case?
If applicable, what's the typical method of finding the best bounds of running time?
I also figured out that T(n) = T(0.8n) + n = O(n).
When solving recurrence relations, the most common way is by repeatly replacing functions by their expressions. S.t. T(n) = T(0.8n) + n = T(0.64n) + 0.8n + n = ... = (1 + 0.8 + 0.64 + 0.512 + ...)n. This is a typical geometric infinite progression. By applying basic calculus, we can easily get that T(n) = 5n = O(n).
When we change the n in the original expression by n^x and x is an arbitrary non-zero constants, we can always let some variable t = n^x, and T(t) is O(t), so n^x should be the same case with n. T(n^x) = O(n^x).
I am trying to find the runtime of the following recurrence using iterative substitution:
T(n) = T(n/2) + T(n/3) + n
The issue is that there are two T(n/x) terms and finding general form for this case has proven to be quite challenging.
Is there a general guideline one should follow using iterative substitution for cases like this?
This recurrence is from the class of Akra–Bazzi recurrences . Following the formula the solution is:
Alternatively, suppose that T(1) = c0 then you can prove that T(n) <= max(6,c0)*n by induction.
You can also use the substitution rule. Here's how:
T(n) = T(n/2)+T(n/3) + n =
= n+(n/2+n/3)+T(n/(2*2))+T(n/(2*3))+T(n/(3*2))+T(n/(3*3))
= n+(n/2+n/3)+(n/(2*2)+n/(2*3)+n/(3*2)+n/(3*3))
+T(n/(2*2*2))+T(n/(2*2*3))
+T(n/(2*3*2))+T(n/(2*3*3))
+T(n/(3*2*2))+T(n/(3*2*3))
+T(n/(3*3*2))+T(n/(3*3*3))=
...
= n * (1 + 5/6 + (5/6)^2 + (5/6)^3 + (5/6)^4 + ...)
= 6 * n (assuming n = 2^k3^k. you get < 6*n otherwise)
Nothing formal here, but
T(n) = 2T(n/2) + n // O(nlog(n))
So your recurrence might still be O(nlog(n))?
Also what is the base case?
In masters theorem were given a "plug-in" formula to find the big O, given it satisfies some condition.
However, what if we have problems like the following below? Can anyone show me how to do a step by step formula. And what topics would help me to know more about these types of questions. Assume that the person asking this question knows nothing about induction.
T(n)=T(n^(1/2))+1
T(n)=T(n-1) + 1
T(n)=T(n-1) + n^c , c is a natural number >1
T(n)= T(n-1) C^n, c is a natural number >1
You'll need to know a little math to do some of these. You can figure out what the recursion looks like when you expand it out all the way to the base case, e.g. for T(n) = T(n-1) + n^c you get T(n) = 1^c + 2^c + ... + n^c, but then you need to know some math in order to know that this is O(n^(c+1)). (The easiest way to see this is by bounding the sum above and below in terms of integrals of x^c). Similarly for T(n) = T(n-1) + c^n you easily get T(n) = c^1 + c^2 + ... + c^n but you again need to use some calculus or something to figure out that this is T(n) = O(c^n).
For T(n) = T(n^(1/2)) + 1 you need to count how many times you apply the recurrence before you get to the base case. Again math helps here. When you take square-root, the logarithm gets cut in half. So you want to know how many times you can cut the logarithm in half until you get to the base case. This is O(log log n).
You can expand upon the formula and work on it:
For example:
T(n) = T(n-1) + 1
T(n) = [T(n-2) + 1] + 1
...
T(n) = 1 + 1 + 1 ... (n times)
So T(n) = O(n).
I want to understand how to arrive at the complexity of the below recurrence relation.
T(n) = T(n-1) + T(n-2) + C
Given T(1) = C and T(2) = 2C;
Generally for equations like T(n) = 2T(n/2) + C (Given T(1) = C), I use the following method.
T(n) = 2T(n/2) + C
=> T(n) = 4T(n/4) + 3C
=> T(n) = 8T(n/8) + 7C
=> ...
=> T(n) = 2^k T (n/2^k) + (2^k - 1) c
Now when n/2^k = 1 => K = log (n) (to the base 2)
T(n) = n T(1) + (n-1)C
= (2n -1) C
= O(n)
But, I'm not able to come up with similar approach for the problem I have in question. Please correct me if my approach is incorrect.
The complexity is related to input-size, where each call produce a binary-tree of calls
Where T(n) make 2n calls in total ..
T(n) = T(n-1) + T(n-2) + C
T(n) = O(2n-1) + O(2n-2) + O(1)
O(2n)
In the same fashion, you can generalize your recursive function, as a Fibonacci number
T(n) = F(n) + ( C * 2n)
Next you can use a direct formula instead of recursive way
Using a complex method known as Binet's Formula
You can use this general approach described here.Please ask if you have more questions.
If you were also interested in finding an explicit formula for T(n) this may help.
We know that T(1) = c and T(2) = 2c and T(n) = T(n-1) + T(n-2) + c.
So just write T(n) and start expanding.
T(n) = T(n-1) + T(n-2) + c
T(n) = 2*T(n-2) + T(n-3) + 2c
T(n) = 3*T(n-3) + 2*T(n-4) + 4c
T(n) = 5*T(n-4) + 3*T(n-5) + 7c
and so on.
You see the coefficients are Fibonacci numbers themselves!
Call F(n) the nth Fibonacci number. F(n) = (phi^n + psi^n)/sqrt(5) where phi = (1+sqrt(5))/2 and psi = -1/phi, then we have:
T(n) = F(n)*2c + F(n-1)*c + (F(n+1)-1)*c
Here is some quick code to demonstrate:
def fib_gen(n):
"""generates fib numbers to avoid rounding errors"""
fibs=[1,1]
for i in xrange(n-2):
fibs.append(fibs[i]+fibs[i+1])
return fibs
F = fib_gen(50) #just an example.
c=1
def T(n):
"""the recursive definiton"""
if n == 1:
return c
if n == 2:
return 2*c
return T(n-1) + T(n-2) + c
def our_T(n):
n=n-2 #just because your intials were T(1) and T(2), sorry this is ugly!
"""our found relation"""
return F[n]*2*c + F[n-1]*c + (F[n+1]-1)*c
and
>>> T(24)
121392
>>> our_T(24)
121392
Is "worse than exponential" accurate enough for your purposes? The special case C=0 defines http://en.wikipedia.org/wiki/Fibonacci_number, which you can see from the article is exponential. Assuming C is positive, your series will be growing faster than this. In fact, your series will lie between the Fibonacci series and a variant of the Fibonacci series in which the golden ratio is replaced by something very slightly larger.
This type of recurrences are called: non-homogeneous recurrence relations and you have to solve in the beginning homogeneous recurrence (the one without a constant at the end). If you are interested, read the math behind it.
I will show you an easy way. Just type your equation in wolfram-alpha and you will get:
So the complexity grows in the same way as either Lucas or Fibonacci number (the bigger of them).
But both of them have the same growth rate:
and therefore your growth rate is an exponential of the golden ratio: O(phi^n)
From what I have studied: I have been asked to determine the complexity of a function with respect to another function. i.e. Given f(n) and g(n), determine O(f(n(). In such cases, I substitute values, compare both of them and arrive at a complexity - using O(), Theta and Omega notations.
However, in the substitution method for solving recurrences, every standard document has the following lines:
• [Assume that T(1) = Θ(1).]
• Guess O(n3) . (Prove O and Ω separately.)
• Assume that T(k) ≤ ck3 for k < n .
• Prove T(n) ≤ cn3 by induction.
How am I supposed to find O and Ω when nothing else (apart from f(n)) is given? I might be wrong (I, definitely am), and any information on the above is welcome.
Some of the assumptions above are with reference to this problem: T(n) = 4T(n/2) + n
, while the basic outline of the steps is for all such problems.
That particular recurrence is solvable via the Master Theorem, but you can get some feedback from the substitution method. Let's try your initial guess of cn^3.
T(n) = 4T(n/2) + n
<= 4c(n/2)^3 + n
= cn^3/2 + n
Assuming that we choose c so that n <= cn^3/2 for all relevant n,
T(n) <= cn^3/2 + n
<= cn^3/2 + cn^3/2
= cn^3,
so T is O(n^3). The interesting part of this derivation is where we used a cubic term to wipe out a linear one. Overkill like that is often a sign that we could guess lower. Let's try cn.
T(n) = 4T(n/2) + n
<= 4cn/2 + n
= 2cn + n
This won't work. The gap between the right-hand side and the bound we want is is cn + n, which is big Theta of the bound we want. That usually means we need to guess higher. Let's try cn^2.
T(n) = 4T(n/2) + n
<= 4c(n/2)^2 + n
= cn^2 + n
At first that looks like a failure as well. Unlike our guess of n, though, the deficit is little o of the bound itself. We might be able to close it by considering a bound of the form cn^2 - h(n), where h is o(n^2). Why subtraction? If we used h as the candidate bound, we'd run a deficit; by subtracting h, we run a surplus. Common choices for h are lower-order polynomials or log n. Let's try cn^2 - n.
T(n) = 4T(n/2) + n
<= 4(c(n/2)^2 - n/2) + n
= cn^2 - 2n + n
= cn^2 - n
That happens to be the exact solution to the recurrence, which was rather lucky on my part. If we had guessed cn^2 - 2n instead, we would have had a little credit left over.
T(n) = 4T(n/2) + n
<= 4(c(n/2)^2 - 2n/2) + n
= cn^2 - 4n + n
= cn^2 - 3n,
which is slightly smaller than cn^2 - 2n.