T(m,n) = 2T(m/2,n)+n, assume T(m,n) is constant if either m<2 or n<2
So what I don't understand is, can this problem be solved using Master Theorem? If so how? If not, is this table correct?
level # of instances size cost of each level total cost
0 1 m, n n n
1 2 m/2, n n 2n
2 4 m/4, n n 4n
i 2^i m/(2^i), n n 2^i * n
k m 1, n n n*m
Thanks!
The Master Theorem might be a little bit of an overkill here and your solution method is not bad (log means logarithmus to base 2, c=T(1,n)):
T(m,n)=n+2T(m/2,n)=n+2n+4T(m/4,n)=n*(1+2+4+..+2^log(m))+2^log(m)*c
=n*(2^(log(m)+1)-1)+m*c=Theta(n*m)
If you use the Master Theorem by treating n as a constant, than you would easily get T(m,n)=Theta(m*C(n)) with a constant C depending on n, but the Master Theorem does not tell you much about this constant C. If you get too smart and inattentive you could easily get burned:
T(m,n)=n+2T(m/2,n)=n*(1+2/nT(m/2,n))=n*Theta(2^(log(m/n)))
=n*Theta(m/n)=Theta(m)
And now, because you left out C(n) in the third step, you got a wrong result!
Related
Update: I'm still looking for a solution without using something from outside sources.
Given: T(n) = T(n/10) + T(an) + n for some a, and that: T(n) = 1 if n < 10, I want to check if the following is possible (for some a values, and I want to find the smallest possible a):
For every c > 0 there is n0 > 0 such that for every n > n0, T(n) >= c * n
I tried to open the function declaration step by step but it got really complicated and I got stuck since I see no advancement at all.
Here is what I did: (Sorry for adding an image, that's because I wrote a lot on word and I cant paste it as text)
Any help please?
Invoking Akra–Bazzi, g(n) = n¹, so the critical exponent is p = 1, so we want (1/10)¹ + a¹ = 1, hence a = 9/10.
Intuitively, this is like the mergesort recurrence: with linear overhead at each call, if we don't reduce the total size of the subproblems, we'll end up with an extra log in the running time (which will overtake any constant c).
Requirement:
For every c > 0 there is n0 > 0 such that for every n > n0, T(n) >= c*n
By substituting the recurrence in the inequality and solving for a, you will get:
T(n) >= c*n
(c*n/10) + (c*a*n) + n >= c*n
a >= 0.9 - (1/c)
Since our desired outcome is for all c (treat c as infinity), we get a >= 0.9. Therefore, smallest value of a is 0.9 which will satisfy T(n) >= c*n for all c.
Another perspective: draw a recursion tree and count the work done per level. The work done in each layer of the tree will be (1/10 + a) as much as the work of the level above it. (Do you see why?)
If (1/10 + a) < 1, this means the work per level is decaying geometrically, so the total work done will sum to some constant multiple of n (whose leading coefficient depends on a). If (1 / 10 + a) ≥ 1, then the work done per level stays the same or grows from one level to the next, so the total work done now depends (at least) on the number of layers in the tree, which is Θ(log n) because the subproblems sizes are dropping by a constant from one layer to the next and that can’t happen more than Θ(log n) times. So once (1 / 10 + a) = 1 your runtime suddenly is Ω(n log n) = ω(n).
(This is basically the reasoning behind the Master Theorem, just applied to non-uniform subproblems sizes, which is where the Akra-Bazzi theorem comes from.)
I'm reading Network Flows - Theory, Algorithms, and Applications and I'm stuck on the proof of the following theorem (Ch. 3, Page 67):
Theorem. Suppose that 𝑧^𝑘 is the objective function value of some solution of a minimization problem at the 𝑘𝑡ℎ iteration of an algorithm and 𝑧^∗ is the minimum objective function value. Furthermore, suppose that the algorithm guarantees that for every iteration 𝑘,
(1) (𝑧^𝑘 − 𝑧^(𝑘+1)) ≥ 𝛼(𝑧^𝑘 − 𝑧^∗)
(i.e., the improvement at iteration 𝑘 + 1 is at least 𝛼 times the total possible improvement)for some constant 𝛼 with 0 < 𝛼 < 1 (which is independent of the problem data). Then the algorithm terminates in 𝑂((𝑙𝑜𝑔𝐻)/𝛼) iterations, where 𝐻 is the difference between the maximum and minimum objective function values.
Proof. The quantity (𝑧^𝑘 − 𝑧^∗) represents the total possible improvement in the objective function value after the 𝑘𝑡ℎ iteration. Consider a consecutive sequence of 2/𝑎 iterations starting from iteration 𝑘. If each iteration of the algorithm improves the objective function value by at least 𝑎(𝑧^𝑘 − 𝑧^∗)/2 units, the algorithm would determine an optimal solution within these 2/𝑎 iterations. Suppose, instead, that at some iteration 𝑞 + 1, the algorithm improves the objective function value by less than 𝑎(𝑧^𝑘 − 𝑧^∗)/2 units. In other words,
(2) 𝑧^𝑞 − 𝑧^(𝑞+1) ≤ 𝑎(𝑧^𝑘 − 𝑧^∗)/2.
The inequality (1) implies that
(3) 𝛼(𝑧^𝑞 − 𝑧^∗) ≤ 𝑧^𝑞 − 𝑧^(𝑞+1)
The inequalities (2) and (3) imply that
(𝑧^𝑞 − 𝑧^∗) ≤ (𝑧^𝑘 − 𝑧^∗)/2,
so the algorithm has reduced the total possible improvement (𝑧^𝑘 − 𝑧^∗) by a factor at least 2. We have thus shown that within 2/𝑎 consecutive iterations, the algorithm either obtains an optimal solution or reduces the total possible improvement by a factor of at least 2. Since 𝐻 is the maximum possible improvement and every objective function value is an integer, the algorithm must terminate within 𝑂((𝑙𝑜𝑔𝐻)/𝛼) iterations.
Why does the author focus on 2/a iterations?
I think it's written this way for computer scientists to understand more easily that we're halving an integer quantity every 2/α rounds, hence the number of 2/α-round super-rounds will be O(log). No particular reason we can't do the math more directly:
T(n) is the gap between the nth solution and an optimal solution.
T(n) ≤ (1 - α) T(n-1) [assumption]
n
T(n) ≤ T(0) (1 - α) [by induction]
-α n x
< T(0) (e ) [by 1 + x < e for x ≠ 0]
-α (ln T(0))/α
T((ln T(0))/α) < T(0) e
= 1,
so (ln T(0))/α rounds suffice.
I was studying the merge-sort subject that I ran into this concept that the number of comparisons in merge-sort (in the worst-case, and according to Wikipedia) equals (n ⌈lg n⌉ - 2⌈lg n⌉ + 1); in fact it's between (n lg n - n + 1) and (n lg n + n + O(lg n)). The problem is that I cannot figure out what these complexities try to say. I know O(nlogn) is the complexity of merge-sort but the number of comparisons?
Why to count comparisons
There are basically two operations to any sorting algorithm: comparing data and moving data. In many cases, comparing will be more expensive than moving. Think about long strings in a reference-based typing system: moving data will simply exchange pointers, but comparing might require iterating over a large common part of the strings before the first difference is found. So in this sense, comparison might well be the operation to focus on.
Why an exact count
The numbers appear to be more detailed: instead of simply giving some Landau symbol (big-Oh notation) for the complexity, you get an actual number. Once you have decided what a basic operation is, like a comparison in this case, this approach of actually counting operations becomes feasible. This is particularly important when comparing the constants hidden by the Landau symbol, or when examining the non-asymptotic case of small inputs.
Why this exact count formula
Note that throughout this discussion, lg denotes the logarithm with base 2. When you merge-sort n elements, you have ⌈lg n⌉ levels of merges. Assume you place ⌈lg n⌉ coins on each element to be sorted, and a merge costs one coin. This will certainly be enough to pay for all the merges, as each element will be included in ⌈lg n⌉ merges, and each merge won't take more comparisons than the number of elements involved. So this is the n⌈lg n⌉ from your formula.
As a merge of two arrays of length m and n takes only m + n − 1 comparisons, you still have coins left at the end, one from each merge. Let us for the moment assume that all our array lengths are powers of two, i.e. that you always have m = n. Then the total number of merges is n − 1 (sum of powers of two). Using the fact that n is a power of two, this can also be written as 2⌈lg n⌉ − 1, and subtracting that number of returned coins from the number of all coins yields n⌈lg n⌉ − 2⌈lg n⌉ + 1 as required.
If n is 1 less than a power of two, then there are ⌈lg n⌉ merges where one element less is involved. This includes a merge of two one-element lists which used to take one coin and which now disappears altogether. So the total cost reduces by ⌈lg n⌉, which is exactly the number of coins you'd have placed on the last element if n were a power of two. So you have to place fewer coins up front, but you get back the same number of coins. This is the reason why the formula has 2⌈lg n⌉ instead of n: the value remains the same unless you drop to a smaller power of two. The same argument holds if the difference between n and the next power of two is greater than 1.
On the whole, this results in the formula given in Wikipedia:
n ⌈lg n⌉ − 2⌈lg n⌉ + 1
Note: I'm pretty happy with the above proof. For those who like my formulation, feel free to distribute it, but don't forget to attribute it to me as the license requires.
Why this lower bound
To proove the lower bound formula, let's write ⌈lg n⌉ = lg n + d with 0 ≤ d < 1. Now the formula above can be written as
n (lg n + d) − 2lg n + d + 1 =
n lg n + nd − n2d + 1 =
n lg n − n(2d − d) + 1 ≥
n lg n − n + 1
where the inequality holds because 2d − d ≤ 1 for 0 ≤ d < 1
Why this upper bound
I must confess, I'm rather confused why anyone would name n lg n + n + O(lg n) as an upper bound. Even if you wanted to avoid the floor function, the computation above suggests something like n lg n − 0.9n + 1 as a much tighter upper bound for the exact formula. 2d − d has its minimum (ln(ln(2)) + 1)/ln(2) ≈ 0.914 for d = −ln(ln(2))/ln(2) ≈ 0.529.
I can only guess that the quoted formula occurs in some publication, either as a rather loose bound for this algorithm, or as the exact number of comparisons for some other algorithm which is compared against this one.
(Two different counts)
This issue has been resolved by the comment below; one formula was originally quoted incorrectly.
equals (n lg n - n + 1); in fact it's between (n lg n - n + 1) and (n lg n + n + O(lg n))
If the first part is true, the second is trivially true as well, but explicitely stating the upper bound seems kind of pointless. I haven't looked at the details myself, but these two statements appear strange when taken together like this. Either the first one really is true, in which case I'd omit the second one as it is only confusing, or the second one is true, in which case the first one is wrong and should be omitted.
I'm currently reading about algorithmic analysis and I read that a certain algorithm (weighted quick union with path compression) is of order N + M lg * N. Apparently though this is linear because lg * N is a constant in this universe. What mathematical operation is being referred to here. I am unfamiliar with the notation lg * N.
The answers given here so far are wrong. lg* n (read "log star") is the iterated logarithm. It is defined as recursively as
0 if n <= 1
lg* n =
1 + lg*(lg n) if n > 1
Another way to think of it is the number of times that you have to iterate logarithm before the result is less than or equal to 1.
It grows extremely slowly. You can read more on Wikipedia which includes some examples of algorithms for which lg* n pops up in the analysis.
I'm assuming you're talking about the algorithm analyzed on slide 44 of this lecture:
http://www.cs.princeton.edu/courses/archive/fall05/cos226/lectures/union-find.pdf
Where they say "lg * N is a constant in this universe" I believe they aren't being entirely literal.
lg*N does appear to increase with N as per their table on the right side of the slide; it just happens to grow at such a slow rate that it can't be considered much else (N = 2^65536 -> log*n = 5). As such it seems they're saying that you can just ignore the log*N as a constant because it will never increase enough to cause a problem.
I could be wrong, though. That's simply how I read it.
edit: it might help to note that for this equation they're defining "lg*N" to be 2^(lg*(N-1)). Meaning that an N value of 2^(2^(65536)) [a far larger number] would give lg*N = 6, for example.
The recursive definition of lg*n by Jason is equivalent to
lg*n = m when 2 II m <= n < 2 II (m+1)
where
2 II m = 2^2^...^2 (repeated exponentiation, m copies of 2)
is Knuth's double up arrow notation. Thus
lg*2= 1, lg*2^2= 2, lg*2^{2^2}= 3, lg*2^{2^{2^2}} = 4, lg*2^{2^{2^{2^2}}} = 5.
Hence lg*n=4 for 2^{16} <= n < 2^{65536}.
The function lg*n approaches infinity extremely slowly.
(Faster than an inverse of the Ackermann function A(n,n) which involves n-2 up arrows.)
Stephen
lg is "LOG" or inverse exponential. lg typically refers to base 2, but for algorithmic analysis, the base usually doesnt matter.
lg n refers to log base n. It is the answer to the equation 2^x = n. In Big O complexity analysis, the base to log is irrelevant. Powers of 2 crop up in CS, so it is no surprise if we have to choose a base, it will be base 2.
A good example of where it crops up is a fully binary tree of height h, which has 2^h-1 nodes. If we let n be the number of nodes this relationship is the tree is height lg n with n nodes. The algorithm traversing this tree takes at most lg n to see if a value is stored in the tree.
As to be expected, wiki has great additional info.
Logarithm is denoted by log or lg. In your case I guess the correct interpretation is N + M * log(N).
EDIT: The base of the logarithm does not matter when doing asymptotic complexity analysis.
Consider a forest implementation of disjoint sets with only the weighted union heuristics (NO PATH COMPRESSION!) with n distinct elements. Define T(n,m) to be the worst case time complexity of executing a sequence of n-1 unions and m finds in any order, where m is any positive integer greater than n.
I defined T(n,m) to be the sequence of doing n-1 unions and then m finds AFTERWARDS because doing the find operation on the biggest tree possible would take the longest. Accordingly, T(n,m) = m*log(n) + n - 1 because each union takes O(1) so n-1 unions is n-1 steps, and each find takes log(n) steps per as the height of the resultant tree after n-1 unions is bounded by log_2 (n).
My problem now is, does the T(n,m) chosen look fine?
Secondly, is T(n,m) Big Omega (m*log(n)) ? My claim is that it is with c = 1 and n >= 2, given that the smallest possible T(n,m) is m*log(2) + 1, which is obviously greater than m*log(2). Seems rather stupid to ask this but it seemed rather too easy for a solution, so I have my suspicions regarding my correctness.
Thanks in advance.
Yes to T(n, m) looking fine, though I suppose you could give a formal induction proof that the worst-case is unions followed by finds.
As for proving that T(n, m) is Ω(m log(n)), you need to show that there exist n0 and m0 and c such that for all n ≥ n0 and all m ≥ m0, it holds that T(n, m) ≥ c m log(n). What you've written arguably shows this only for n = 2.