I'm not sure about the O() complexity of these functions. My answers are in the boxes. Someone told me they are all O(n) but I don't understand why that is. Thanks.
All four are O(n) (ignoring that the two best case questions should use Ω(n)) since you must examine every node.
Consider height: you have to recursively check each subtree, terminating only once you reach the bottom of a tree. That means you're going to reach every leaf node eventually. You can't terminate early.
The same goes for balanced; you can't verify that a tree is balanced without first verifying that each subtree is balanced, which in this implementation means calling height for each subtree.
Now for the wording of the exam. Big O notation is used for worst cases because a worst case is (by definition) "bigger" than all other cases. An upper bound for the worst case is necessarily an upper bound for all cases. Similarly, a best case is by definition "smaller" than all other cases. An upper bound on the best case is mostly useless, because you can't say anything about the remaining cases.
When talking about best cases, you use Ω (big omega) notation, which provides a lower bound. Saying the best case is Ω(n) tells you that no matter how good the best base (and thus every case) is, it's no smaller than n.
For height and `balanced, you can actually show that the best case is Ω(n) and the worst case is O(n). In that case, you can combine them and say that each is Θ(n); the upper and lower bounds match.
height()
Best case: both left and right trees are null. Therefore O(1) for a single max comparison, though technically, n = 1, so you can say O(n).
Worst case: must completely traverse both left and right trees when neither is null. O(n)
Same for balanced(), as far as I can tell.
Related
One way to find the lower bound of a comparison based algorithm is to use the decision tree. U have two questions regarding this method :
1) We know that the height of the tree is path that connects the root node to the farthest leaf node ( longest path) which is equal to the number of comparisons made from the root node to the leaf node. Therefore , when we draw the tree for a comparison based algorithm we simply need to find the worst case time which corresponds to the largest path and therefore corresponds to the height of the tree. Now for any tree the height<=log2(number of leaf nodes) which is identical to Cworst(n) <= log2(n) and now we have a lower bound for the Cworst(n) and therefore the lower bound of the problem = log2(n). Is my understanding right ?
2) What is the meaning of having an inequality for Cworst(n) for a specific comparison problem? Does this means that for a specific comparison problem we can draw many trees and every time for the path of the worst case scenario the height will have a value that satisfies the equality ? This means that for a specific problem we can draw many different trees ?
A decision tree illustrates the possible executions of an algorithm on a specific category of inputs. In the case of comparison-based sorting, a category would consist of all input lists of a certain size, so there's one tree for n = 1, one for n = 2, one for n = 3, and so on. Each tree is agnostic to the exact input values, but traces the possible directions the computation might go in depending on how the input values compare to each other.
One application of a decision tree is to reason about upper and lower bounds on the runtime complexity. As you mentioned, the height of a tree represents the upper bound on the runtime for that tree's input size, and if you can find a general relation between the input size and the tree height for all the decision trees of an algorithm, you have found an expression for the upper bound on the runtime. For example, if you analyze Bubble Sort, you will find that the decision tree for input size n has a height that's roughly n * (n + 1) / 2, so now you know that its runtime is bounded by O(n^2). And since we have an upper bound on the general runtime, that also becomes an upper bound on the worst-case runtime.
When looking at a specific algorithm, we're usually interested in how fast it could possibly be (best case), how fast it usually is (average case), and how slow it could possibly be (worst case). And we often express the best case using a lower bound (Omega), because a lower bound for the best case is also a lower bound for the general runtime; similarly, we often express the worst case using an upper bound (O), because an upper bound for the worst case is also an upper bound for the general runtime. But we don't have to - O, Omega, and Theta are only mathematical tools that say something about a function, not caring that the function in our case describes a runtime complexity. So let's do something unusual: let's look at all the possible algorithms for a problem, and use decision trees to try to figure something out about all their worst-case complexities. Then, the interesting question isn't what the upper bound is, because it's easy to make extremely slow sorting algorithms. Instead, we're interested in the lower bound: what's the best worst case? Which algorithm makes the best guarantee about how slow it will be in the worst case?
Any sorting algorithm must be able to handle any order of its input elements. Each leaf node represents one particular final permutation (rearrangement) of the input elements, and with input size n, there are n! permutations. So a decision tree for input size n has at least n! leaf nodes. Any algorithm that wants to have a good worst case needs to have a balanced tree where all the leaf nodes are on the deepest or second-deepest level. And a balanced tree with n! leaf nodes must have a height of at Omega(n lg n). Now we know something very interesting: for any comparison-based sorting algorithm, the best possible height (which represents the worst-case runtime) is at least n lg n! In other words, it is impossible to create a comparison-based sorting algorithm that always is faster than n lg n.
(Note: height <= log2(leaf nodes) is only the case for balanced trees. A tree's eight might be as much as the number of nodes minus one.)
What is the relation/difference between worst case time complexity of an algorithm and its upper bound?
The term "upper bound" is not very clear, as it may refer to two possible things:
The upper bound of the algorithm - the bound where the algorithm can never run "slower" than it. This is basically the worst case performance of it, so if this is what you mean - the answer is pretty simple.
big-O notation, which provides an upper bound on the complexity of the algorithm under a specific analysis. The big-O notation is a set of functions, and can be applied to any analysis of an algorithm, including worst case, average case, and even best case.
Let's take Quick Sort as an example.
Quick Sort is said to have O(n^2) worst case performance, and O(nlogn) average case performance. How can one algorithm has two complexities? Simple, the function representing the analysis of average case, and the one representing the worst case, are completely different funcitons - and we can apply big O notation on each of them, there is no restriction about it.
Moreover, we can even apply it to best-case. Consider a small optimization to quick-sort, where it first checks if the array is already sorted, and if it is - it stops immidiately. This is effectively O(n) operation, and there is some input that will provide this behavior - so we can now say that the algorithm's best case complexity is O(n)
The difference between worst case and big O(UPPER BOUND) is that
the worst case is a case that actually happens to your code,
the upper bound is an overestimate, an assumption that we put in order to
calculate the big O, it doesn't have to happen
example on insertion sort:
Worst Case:
The numbers are all arranged reversely so you need to arrange and move every single
number
Pseudo-code
for j=2 to n
do key = a[i]
i=j-1
while i>0 & a[i]>key
do a[i+1] = a[i]
i=i-1
end while
a[i+1]=key
end for
Upper Bound:
We assume that the order of the inner loop is i =n-1 every single time, but in fact,
it is changeable every time, it can't be n-1 every time, but we assumed
/overestimated it to calculate the big O
The following is a homework assignment, so I would rather get hints or bits of information that would help me figure this out, and not complete answers.
Consider S an algorithm solution to a problem that takes as input an array A of size n. After analysis, the following conclusion was obtained:
Algorithm S executes an O(n)-time computation for each even number in A.
Algorithm S executes an O(logn)-time computation for each odd number in A.
What are the best and worst case time for algorithm S?
From this I understand that the time complexity changes in accordance to n being even or odd. In other words, if n is even, S takes O(n) time and when n is odd, S takes O(logn).
Is it a simple matter of taking the best case and the worst case of both growth-rates, and choosing their boundaries? Meaning:
Best case of O(n) is O(1), and worst case is O(n).
Best case of O(logn) is O(logn) and worst case is O(logn).
Therefore the best case for Algorithm S is O(logn) and the worst case is O(n)?
Am I missing something? or am I wrong in assessing the different best/worst case of both cases of big-Oh?
1st attempt:
Ok, so I completely misunderstood the problem. Thanks to candu, I can now better understand what is required of me, and so try to calculate the best and worst case better.
It seems that Algorithm S changes its runtime according to EACH number in A. If the number is even, the runtime is O(n), and if the number is odd, we get O(logn).
The worst case will be composed of an array A of n even numbers, and for each the algorithm will run O(n). In other words, the worst case runtime for Algorithm S should be n*O(n).
The best case will be composed of an array A of n odd numbers, and for each the algorithm will run O(logn). The best case runtime for algorithm S should be n*O(logn).
Am I making any sense? is it true then that:
Best case of algorithm S is nO(logn) and worst case is nO(n)?
If that is true, can it be rewritten? for example, as O(log^n(n)) and O(n^n)? or is this an arithmetic mistake?
2nd attempt:
Following JuanLopes' response, it seems like I can rewrite nO(n) as O(n*n) or O(n^2), and nO(logn) as O(nlogn).
Does it make sense now that Algorithm S runs at O(nlogn) at the best case, and O(n^2) at the worst case?
There's a bit of confusion here: the algorithm runtime doesn't depend on n being even or odd, but on whether the numbers in A are even or odd.
With that in mind, what sort of input A would make Algorithm S run faster? Slower?
Also: it doesn't make sense to say that the best case of O(n) is O(1). Suppose I have an algorithm ("Algorithm Q") that is O(n); all I can say is that there exists a constant c such that, for any input of size n, Algorithm Q takes less than cn time. There is no guarantee that I can find specific inputs for which Algorithm Q is O(1).
To give a concrete example, this takes linear time no matter what input it is passed:
def length(A):
len = 0
for x in A:
len += 1
return len
A few thoughts.
First, there is no mention of asymptotically tight time. So an O(n) algorithm can actually be an O(logn) one. So just imagine the best case running time this algorithm can be in this case. I know, this is a little picky. But this is a homework, I guess it's always welcome to mention all the possibilities.
Second, even if it's asymptotically tight, it doesn't necessarily mean it's tight for all elements. Consider insertion sort. For each new element to insert, we need to find the correct position in the previous already-sorted subarray. The time is proportional to the number of element in subarray, which has the upper bound O(n). But it doesn't mean each new element need exactly #n comparisons to insert. Actually, the shorter the subarray, the quicker the insertion.
Back to this question. "executes an O(logn)-time computation for each odd number in A." Let's assume all odd nubmers. It could be that the first odd takes O(log1), the second odd takes O(log2), .. the nth takes O(logn). Totally, it takes O(logn!). It doesn't contradicts "O(logn) for each odd number".
As to worst case, you may analysize it in much the same way.
So I have seen this question about probabilistic skip list space consumption: (answer)
but I think that the asker wasn't clear if he wanted an expected approach or the worst case approach.
So I would like to raise this question into a debate again and I will explain why I'm confused.
To be clear - I'm looking for the space complexity of a probabilistic skip list in the worst case.
This is what I had in mind:
On one hand, we assume that the maximum levels number is log(n) it's easy to infer that in the worst case we might have n nodes in each level which will give us O(nlogn).
On the other hand, I assume that there might be more than log(n) levels (e.g lists) and we set that bound to be n - then we get nn => O(n^2)
BUT! I don't understand why we have the right to limit the levels, if a new level depends on the coin toss, let's assume that in the worst case we will get infinte times Heads (which means a new level) and then we difer that it's not even bounded?!
I'm confused.
If you don't set an upper bound on the height of the skip list, then there is no upper-bound to the worst-case space usage. For any bound you could place, there's some horribly unlucky and astronomically improbable execution of the skiplist that would result in a number of layers so high that the upper bound wouldn't hold. For this reason, in this case there isn't an upper bound on the space usage.
That said, most standard skiplist implementations place some upper bound M on the height of each node, usually chosen so that M is going to be bigger than log n. In that case, the worst-case space usage would be Θ(Mn), which happens if every mode uses all M levels
Hope this helps!
There is a question on an assignment that was due today which solutions have been released for, and I don't understand the correct answer. The question deals with best-case performance of disjoint sets in the form of disjoint set forests that utilize the weighed union algorithm to improve performance (the smaller of the trees has its root connected as a child to the root of the larger of the two trees) but without using the path compression algorithm.
The question is whether the best case performance of doing (n-1) Union operations on n singleton nodes and m>=n Find operations in any order is Omega(m*logn) which the solution confirms is correct like this:
There is a sequence S of n-1 Unions followed by m >= n Finds that takes Omega(m log n) time. The sequence S starts with a sequence n-1 Unions that builds a tree with depth Omega(log n). Then it has m>=n Finds, each one for the deepest leaf of that tree, so each one takes
(log n) time.
My question is, why does that prove the lower bound is Omega(m*logn) is correct? Isn't that just an isolated example of when the bound would be Omega(m*logn) that doesn't prove it for all inputs? I am certain one needs to only show one counter-example when disproving a claim but needs to prove a predicate for all possible inputs in order to prove its correctness.
In my answer, I pointed out the fact that you could have a case when you start off by joining two singleton nodes together. You then join in another singleton to that 2-node tree with 3 nodes sharing the same parent, then another etc., until you join together all the n nodes. You then have a tree where n-1 nodes all point up to the same parent, which is essentially the result you obtain if you use path compression. Then every FIND is executed in O(1) time. Thus, a sequence of (n-1) Unions and m>=n Finds ends up being Omega(n-1+m) = Omega(n+m) = Omega(m).
Doesn't this imply that the Omega(m*logn) bound is not tight and the claim is, therefore, incorrect? I'm starting to wonder if I don't fully understand Big-O/Omega/Theta :/
EDIT : fixed up the question to be a little clearer
EDIT2: Here is the original question the way it was presented and the solution (it took me a little while to realize that Gambarino and the other guy are completely made up; hardcore Italian prof)
Seems like I indeed misunderstood the concept of Big-Omega. For some strange reason, I presumed Big-Omega to be equivalent to "what's the input into the function that results in the best possible performance". In reality, most likely unsurprisingly to the reader but a revelation to me, Big-Omega simply describes the lower bound of a function. That's it. Therefore, a worst case input will have a lower and upper bounds (big-O and omega), and so will the best possible input. In case of big-omega here, all we had to do was come up with a scenario where we pick the 'best' input given the limitations of the worst case, i.e. that there is some input of size n that will take the algorithm at least m*logn steps. If such input exists, then the lower bound is tight.