I've been viewing some video lectures from MIT's opencourseware website, and on the third lecture video the lecturer goes over Recursive Matrix Multiplication and comes up with the time complexity being:
T(n) = Θ(n3)
It's obvious to me that I really need to review some of my math, but I really don't see the connection from that answer and any one of the cases mentioned for the Master Theorem Method. I know that the recursive relation is of the form:
T(n) = a*T(n/b) + f(n) for n > 1.
With this recursive relation: a = 8, b = 2, and f(n) = Θ(n2).
So, how did they get Θ(n3)?
He mentioned that log28 = 3, which makes sense. But, I just can't figure out which of the three cases that the example recursive relation corresponds to, using the value of f(n).
Since, Case 2 is the only one where f(n) = Θ(anything), then I'm guessing that that's it. Yet, I guess my problem relates to properties of logarithms or exponents.
If f(n) = Θ(nlog28 * (log2n)k+1) then how do you go from having a Θ(n3) for f(n) to a Θ(n2) using case 2? What is it that I might be missing here?
Check out the Wiki page on the Master Theorem.
They discuss this very exact situation (a = 8, b=2, f(n) = O(n2)) when discussing case 1. (not case 2). Note that if f(n) = Θ(n2) then f(n) = O(n2), so case 1 can be applied.
Hope that helps.
Related
What is the value of following recurrence:
T(n) = T(n/4) + T(n/2) + cn², T(1) = c, T(0) = 0
Where c is a positive constant:
T(n) = O(n³)
T(n) = O(n²)
T(n) = O(n² log n)
T(n) = O(n log n)
The correct answer is 2, but I have a doubt. According to the definition of O(f(n)) it gives us an upper bound, the O(n²) is the least upper bound. So in my opinion O(n³) and O(n² log n) should also be true.
Let
T(n) = 1/2n² + 3n
Which of the following statements are true (Check all that apply.)
T(n) = O(n)
T(n) = Ω(n)
T(n) = θ(n²)
T(n) = O(n³)
Here, the correct answers are 2, 3, and 4.
So, am I understanding the definition incorrectly or am I making some mistake?
Let's try to prove for the first recurrence using induction.
I'll use this definition of Big O (from CLRS):
for some constants and .
Basis Step:
. Same for , but since , it doesn't contribute anything to the recurrence and we can choose as our base case.
Inductive Step:
Assume , now deduce for . Since , our assumption implies .
We have
for some constant .
Since
holds, setting and to all the inductively assumed expressions yields:
(If I screwed up somewhere, let me know!)
So in my opinion O(n³) and O(n² log n) should also be true.
, so yes, you are entirely correct. You can prove it correct as done above. Informally, however, people often use interchangeably with tight upper bound. It's imprecise, but customary. If you got those questions from a facility of higher learning, that's problematic, of course.
So, am I understanding the definition incorrectly or am I making some
mistake?
You are basically correct. The real world isn't about formal correctness, though, so beware of informalisms and know better. As an aside, many people also treat the Landau symbols , , and the same, even though they absolutely aren't.
Solve: T(n)=T(n-1)+T(n/2)+n.
I tried solving this using recursion trees.There are two branches T(n-1) and T(n/2) respectively. T(n-1) will go to a higher depth. So we get O(2^n). Is this idea correct?
This is a very strange recurrence for a CS class. This is because from one point of view: T(n) = T(n-1) + T(n/2) + n is bigger than T(n) = T(n-1) + n which is O(n^2).
But from another point of view, the functional equation has an exact solution: T(n) = -2(n + 2). You can easily see that this is the exact solution by substituting it back to the equation: -2(n + 2) = -2(n + 1) + -(n + 2) + n. I am not sure whether this is the only solution.
Here is how I got it: T(n) = T(n-1) + T(n/2) + n. Because you calculate things for very big n, than n-1 is almost the same as n. So you can rewrite it as T(n) = T(n) + T(n/2) + n which is T(n/2) + n = 0, which is equal to T(n) = - 2n, so it is linear. This was counter intuitive to me (the minus sign here), but armed with this solution, I tried T(n) = -2n + a and found the value of a.
I believe you are right. The recurrence relation will always split into two parts, namely T(n-1) and T(n/2). Looking at these two, it is clear that n-1 decreases in value slower than n/2, or in other words, you will have more branches from the n-1 portion of the tree. Despite this, when considering big-o, it is useful to just consider the 'worst-case' scenario, which in this case is that both sides of the tree decreases by n-1 (since this decreases more slowly and you would need to have more branches). In all, you would need to split the relation into two a total of n times, hence you are right to say O(2^n).
Your reasoning is correct, but you give away far too much. (For example, it is also correct to say that 2x^3+4=O(2^n), but that’s not as informative as 2x^3+4=O(x^3).)
The first thing we want to do is get rid of the inhomogeneous term n. This suggests that we may look for a solution of the form T(n)=an+b. Substituting that in, we find:
an+b = a(n-1)+b + an/2+b + n
which reduces to
0 = (a/2+1)n + (b-a)
implying that a=-2 and b=a=-2. Therefore, T(n)=-2n-2 is a solution to the equation.
We now want to find other solutions by subtracting off the solution we’ve already found. Let’s define U(n)=T(n)+2n+2. Then the equation becomes
U(n)-2n-2 = U(n-1)-2(n-1)-2 + U(n/2)-2(n/2)-2 + n
which reduces to
U(n) = U(n-1) + U(n/2).
U(n)=0 is an obvious solution to this equation, but how do the non-trivial solutions to this equation behave?
Let’s assume that U(n)∈Θ(n^k) for some k>0, so that U(n)=cn^k+o(n^k). This makes the equation
cn^k+o(n^k) = c(n-1)^k+o((n-1)^k) + c(n/2)^k+o((n/2)^k)
Now, (n-1)^k=n^k+Θ(n^{k-1}), so that the above becomes
cn^k+o(n^k) = cn^k+Θ(cn^{k-1})+o(n^k+Θ(n^{k-1})) + cn^k/2^k+o((n/2)^k)
Absorbing the lower order terms and subtracting the common cn^k, we arrive at
o(n^k) = cn^k/2^k
But this is false because the right hand side grows faster than the left. Therefore, U(n-1)+U(n/2) grows faster than U(n), which means that U(n) must grow faster than our assumed Θ(n^k). Since this is true for any k, U(n) must grow faster than any polynomial.
A good example of something that grows faster than any polynomial is an exponential function. Consequently, let’s assume that U(n)∈Θ(c^n) for some c>1, so that U(n)=ac^n+o(c^n). This makes the equation
ac^n+o(c^n) = ac^{n-1}+o(c^{n-1}) + ac^{n/2}+o(c^{n/2})
Rearranging and using some order of growth math, this becomes
c^n = o(c^n)
This is false (again) because the left hand side grows faster than the right. Therefore,
U(n) grows faster than U(n-1)+U(n/2), which means that U(n) must grow slower than our assumed Θ(c^n). Since this is true for any c>1, U(n) must grow more slowly than any exponential.
This puts us into the realm of quasi-polynomials, where ln U(n)∈O(log^c n), and subexponentials, where ln U(n)∈O(n^ε). Either of these mean that we want to look at L(n):=ln U(n), where the previous paragraphs imply that L(n)∈ω(ln n)∩o(n). Taking the natural log of our equation, we have
ln U(n) = ln( U(n-1) + U(n/2) ) = ln U(n-1) + ln(1+ U(n/2)/U(n-1))
or
L(n) = L(n-1) + ln( 1 + e^{-L(n-1)+L(n/2)} ) = L(n-1) + e^{-(L(n-1)-L(n/2))} + Θ(e^{-2(L(n-1)-L(n/2))})
So everything comes down to: how fast does L(n-1)-L(n/2) grow? We know that L(n-1)-L(n/2)→∞, since otherwise L(n)∈Ω(n). And it’s likely that L(n)-L(n/2) will be just as useful, since L(n)-L(n-1)∈o(1) is much smaller than L(n-1)-L(n/2).
Unfortunately, this is as far as I’m able to take the problem. I don’t see a good way to control how fast L(n)-L(n/2) grows (and I’ve been staring at this for months). The only thing I can end with is to quote another answer: “a very strange recursion for a CS class”.
I think we can look at it this way:
T(n)=2T(n/2)+n < T(n)=T(n−1)+T(n/2)+n < T(n)=2T(n−1)+n
If we apply the master's theorem, then:
Θ(n∗logn) < Θ(T(n)) < Θ(2n)
Remember that T(n) = T(n-1) + T(n/2) + n being (asymptotically) bigger than T(n) = T(n-1) + n only applies for functions which are asymptotically positive. In that case, we have T = Ω(n^2).
Note that T(n) = -2(n + 2) is a solution to the functional equation, but it doesn't interest us, since it is not an asymptotically positive solution, hence the notations of O don't have meaningful application.
You can also easily check that T(n) = O(2^n). (Refer to yyFred solution, if needed)
If you try using the definition of O for functions of the type n^a(lgn)^b, with a(>=2) and b positive constants, you see that this is not a possible solution too by the Substitution Method.
In fact, the only function that allows a proof with the Substitution Method is exponential, but we know that this recursion doesn't grow as fast as T(n) = 2T(n-1) + n, so if T(n) = O(a^n), we can have a < 2.
Assume that T(m) <= c(a^m), for some constant c, real and positive. Our hypothesis is that this relation is valid for all m < n. Trying to prove this for n, we get:
T(n) <= (1/a+1/a^(n/2))c(a^n) + n
we can get rid of the n easily by changing the hypothesis by a term of lower order. What is important here is that:
1/a+1/a^(n/2) <= 1
a^(n/2+1)-a^(n/2)-a >= 0
Changing variables:
a^(N+1)-a^N-a >= 0
We want to find a bond as tight as possible, so we are searching for the lowest a possible. The inequality we found above accept solutions of a which are pretty close to 1, but is a allowed to get arbitrarily close to 1? The answer is no, let a be of the form a = (1+1/N). Substituting a at the inequality and applying the limit N -> INF:
e-e-1 >= 0
which is a absurd. Hence, the inequality above has some fixed number N* as maximum solution, which can be found computationally. A quick Python program allowed me to find that a < 1+1e-45 (with a little extrapolation), so we can at least be sure that:
T(n) = ο((1+1e-45)^n)
T(n)=T(n-1)+T(n/2)+n is the same as T(n)=T(n)+T(n/2)+n since we are solving for extremely large values of n. T(n)=T(n)+T(n/2)+n can only be true if T(n/2) + n = 0. That means T(n) = T(n) + 0 ~= O(n)
I know that we can apply the Master Theorem to find the running time of a divide and conquer algorithm, when the recurrence relation has the form of:
T(n) = a*T(n/b) + f(n)
We know the following :
a is the number of subproblems that the algorithm divides the original problem
b is the size of the sun-problem i.e n/b
and finally.. f(n) encompasses the cost of dividing the problem and combining the results of the subproblems.
Now we then find something (I will come back to the term "something")
and we have 3 cases to check.
The case that f(n) = O(n^log(b)a-ε) for some ε>0; Then T(n) is O(n*log(b)a)
The case that f(n) = O(n^log(b)a) ; Then T(n) is O(n^log(b)a * log n).
If n^log(b)a+ε = O(f(n)) for some constant ε > 0, and if a*f(n/b) =< cf(n) for some constant
c < 1 and almost all n, then T(n) = O(f(n)).
All fine, I am recalling the term something. How we can use general examples (i.e uses variables and not actual numbers) to decide which case the algorithm is in?
In instance. Consider the following:
T(n) = 8T(n/2) + n
So a = 8, b = 2 and f(n) = n
How I will proceed then? How can I decide which case is? While the function f(n) = some big-Oh notation how these two things are comparable?
The above is just an example to show you where I don't get it, so the question is in general.
Thanks
As CLRS suggests, the basic idea is comparing f(n) with n^log(b)a i.e. n to the power (log a to the base b). In your hypothetical example, we have:
f(n) = n
n^log(b)a = n^3, i.e. n-cubed as your recurrence yields 8 problems of half the size at every step.
Thus, in this case, n^log(b)a is larger because n^3 is always O(n) and the solution is: T(n) = θ(n^3).
Clearly, the number of subproblems vastly outpaces the work (linear, f(n) = n) you are doing for each subproblem. Thus, the intuition tells and master theorem verifies that it is the n^log(b)a that dominates the recurrence.
There is a subtle technicality where the master theorem says that f(n) should be not only smaller than n^log(b)a O-wise, it should be smaller polynomially.
The Master Method is a direct way to get the solution. The Master Method works only for following type of recurrences or for recurrences that can be transformed to following type.
T(n) = a T(n / b) + f(n) where a ≥ 1, b > 1, and f(n) = Θ(nc).
There are following three cases:
If c < logb(a) then T(n) = Θ(nlogb(a)).
If c = logb(a) then T(n) = Θ(nc log(n)).
If c > logb(a) then T(n) = Θ(f(n)).
In the Master Method, if f(n) contains some term of log(n), is it possible to solve this by the Master Method?
for example in
T(n)=4T(n/2)+n^2logn
Here master's theorem appplicable or not
It is not really possible to tell directly whether or not the Master Method works for some logarithmic function. This would depend on the specific recurrence you're trying to solve. It all depends on how f grows in comparison to nlogb a.
In the example given by JPC (where T(n) = 4T(n/2) + log(n)), it is indeed possible. However, also consider the example T(n) = 2T(n/5) + log(n). In this recurrence it is harder to determine whether nlog5 2 grows faster than log(n). If the logarithmic function f(n) gets more complex (e.g. log3(n/2)), it becomes even harder.
In short, it may be hard to determine for logarithmic functions how they grow when compared to an exponential function when the exponent is less than 1 (for exponents >= 1, log(n) is always faster). If it doesn't seem to work for you, you'll have to use other techniques to solve the recurrence.
I understand big-oh and theta. the question is as follows,
prove or disprove: f(n) = theta(g(n) => h(f(n)) = O(h(g(n))) if h(n) is an increasing function. h(n1) > h(n2) when n1 > n2
So, in the above question, I am stuck at the point of understanding the increasing function. If I am trying to find some function to disprove it, say eg., n and 2n is this acceptable? becos big-oh represents rapidly growing and not just by a constant factor, but there is no such condition with h(n) function defined (I mean, I consider > as literally greater than here, is that wrong?)
Also, even if I find something like h(f(n)) growing at the same rate as h(g(n)) which means they are theta essentially, are they still big-oh. becos loose bound of theta is big-oh in that case, I can never disprove the above statement.
Please correct me if I my understanding deviated at some point while going thru the sequence. Thanks!
I always start problems like this by re-reading and then writing down the definitions of Theta(f) and O(f). If you can find a simple example which disproves the assertion, you won't have to prove anything. I like the idea of very rapidly growing h(x) such as h(x) = 2^x. Then if you have e.g. f(x) = 2x, g(x) = x, h(f(x)) = 2^(2x), h(g(x)) = 2^x and h(f(x)) / h(g(x)) = 2^x - try pluging these into your definitions and see what happens. Notice that, since f(x) = 2g(x), at least by my definition, you have f(x) = Theta(g(x)) and g(x) = Theta(f(x))
I found a contradictory example to solve this problem, by taking f(n) = 2n and g(n) = n which satisfies the given f(n) = theta(g(n)).
Now, let h(x) = 2^x, this implies that h(f(n)) = 2^2n = 4^n and h(g(n)) = 2^n.
By definition, this means that h(g(n)) = O(h(f(n))). Hence disproved.
I never knew that contradictory examples were sufficient for disproving. I was trying a more formal way to prove it and was getting messed up with details.
Thanks.