How does a program's worst case or average case dependent on log function? How does the base of log come in play?
The log factor appears when you split your problem to k parts, of size n/k each and then "recurse" (or mimic recursion) on some of them.
A simple example is the following loop:
foo(n):
while n > 0:
n = n/2
print n
The above will print n, n/2, n/4, .... , 1 - and there are O(logn) such values.
the complexity of the above program is O(logn), since each printing requires constant amount of time, and number of values n will get along the way is O(logn)
If you are looking for "real life" examples, in quicksort (and for simplicity let's assume splitting to exactly two halves), you split the array of size n to two subarrays of size n/2, and then you recurse on both of them - and invoke the algorithm on each half.
This makes the complexity function of:
T(n) = 2T(n/2) + O(n)
From master theorem, this is in Theta(nlogn).
Similarly, on binary search - you split the problem to two parts, and recurse only on one of them:
T(n) = T(n/2) + 1
Which will be in Theta(logn)
The base is not a factor in big O complexity, because
log_k(n) = log_2(n)/log_2(k)
and log_2(k) is constant, for any constant k.
Related
For this pseudocode, what would be big O of this:
BinarySum(A, i, n) {
if n=1 then
return A[i] // base case
return BinarySum(A, i, n/2) + BinarySum(A, i+[n/2], [n/2])
}
This is really confusing because I think it is O(logn) since you are dividing by 2 each time you run the function but then some people I spoke to think it is O(n) and not O(logn) because the algorithm doesn't half the problem by picking and choosing one half of the array. What is it?
TL;DR
The runtime is θ(n).
How to determine runtime
The recurrence relation for the algorithm is
T(n) = 2T(n/2) + O(1)
Because we have two recursive calls for half of the array in every call and need constant time in a single call. BTW: this is the same recurrence relation which also describes a binary tree traversal.
You can use the Master-theorem to determine the runtime here since a >= 1 and b > 1 and the recurrence relation has the form
T(1) = 1; T(n) = aT(n/b) + f(n)
This is case one of the theorem meaning f(n) = O(n ^ logb(a)). This is because with a = 2, b = 2 and f(n) = O(1) like in this case log2(2) = 1 and therefore f(n) = O(1) = O(n¹) = O(n).
When case one is applicable the Master-theorem says that the runtime of the algorithm is θ(n).
First, let me say that I like the other answer that links the recurrence with the traversal of binary trees. For a balanced binary tree, this is indeed the recurrence, and so the complexity must necessarily be the same as a depth-first traversal, which we all know is O(n). I like this analogy because it clearly says that the result doesn't just apply to the recurrence T(n) = 2T(n/2) + O(1) but to anything where you split the input into chunks of sizes m[0], m[1], ... that sum to size n and do T(n) = T(m[0]) + T(m[1]) + T(m[2]) + ... + O(1). You don't have to split the input into two equally sized parts to get O(n); you just have to spent constant time and then recurse on disjoint parts of the input.
Using the Master's Theorem, I feel, is a bit overkill for this one. It is a big gun, and it gives us the correct answer, but if you are like me, it doesn't give you much intuition about why the answer is what it is. With this particular recurrence, we can get the correct answer and an intuitive understanding of it with a few drawings.
We can break down what happens at each level of the recursion and maybe draw it like this:
We have the work that we are handling on the left, i.e., the size of the input and the actual time we spend at each function call on the right. We have input size n at the first level, but we only spend one "computation unit" of time on it. We have two chunks of size n/2 that we spend two units of time on at the next level. At the next level, we have four chunks, each of size n/4, and we spend four units on them. This continues until our chunks have size one, and we have n of those, that we spend n units of time on.
The total time we spend is the area of the red blocks on the right. The depth of the recursion is O(log n) but we don't need to worry about that to analyse the time. We will just flip the "time" bit and look at it this way:
The total time we spend must be n for the original bottom layer (now top layer), n/2 for the next layer, n/4 for the next, and so on. Move n outside of parentheses, and all we have to worry about is the sum 1+1/2+1/4+1/8+.... The sum ends at some point, of course. We only have O(log n) terms. But we don't have to worry about that at all, because even if it continued forever we wouldn't sum to more than two.
You might recognise this as a geometric series. They have the form sum x^i for i=0 to infinity, and when |x|<1 they converge to 1/(1-x). Proving this takes a little calculus, but when x = 1/2 as we have, it is easy to draw the series and get the result from there.
Take the size n layer and then start putting the remaining layers next to each other under it. First, you put down the n/2 layer. It takes half of the space. Then you put the n/4 layer next to it, and it takes half of the remaining space. The n/8 layer will take half of the remaining space, the n/16 layer will take half of the remaining space, and it will continue like this as if it were a reenactment of Zeno's paradox.
If you keep taking half of what is left forever, you can never get more than you started with, so adding up all the layers except the first cannot give you more time spent than you spent on the very first layer. The total time you would do if you kept recursing forever (and time worked like real numbers) would be linear. Since you stop before forever, it is still going to be linear. Infinity gives us O(n) so recursion depth O(log n) will as well.
Of course, getting O(n) from observing that T(n) < T'(n) = O(n) where T'(n) continues subdividing forever only tells us that T(n) = O(n) and not that T(n) = Omega(n), there you have to show that you don't spend substantially less time than n, but considering that the largest layer is n, it should be obvious that the recursion also runs in Omega(n).
If you don't cut the data size in half every recursion, but cut the data in some other chunks that add up to n, you still get O(n) of course--think of traversing a tree--but it gets a hell of a lot harder to draw, and I've never managed to come up with a good illustration of that. For splitting the data in half, though, the drawing is simple and the conclusions we draw from it gives us the correct running time.
The same drawing also tells us that the recurrence T(n) = T(n/2) + O(n) is in O(n). Here, we don't have to flip the first drawing, because we start out with the largest layer on top. We spend time n then n/2 then n/4 and so on, and we end up spending 2n time units. Because 2 isn't special here, we have T(n) = T(f·n) + O(n) = O(n) for any fraction 0 ≤ f < 1, it is just a lot harder to draw when f ≠ 1/2.
First a bug:
BinarySum(A, i, n) {
if n=1 then
return A[i] // base case
return BinarySum(A, i, n/2) + BinarySum(A, i+(n/2), (n - n/2))
// ^^^^
}
The second half might be of uneven length. And then the last value was dropped for n/2.
Recursively this might be several values.
On the complexity. Having A[0] + A[1] + A[2] + ... + A[n-1].
The recursion goes down to a single A[i] and for every + above adds exactly 1 left and right. So (n-1 subtrees + n leafs = 2n-1) O(n). Furthermore the call tree is irrelevant. BinarySum is not faster (than non-binary sums) unless using multithreading.
There's a difference between making just one recursive function call, like binary search does, and two recursive function calls, like your code does. In both cases, the maximum depth of the recursion is O(log n), but the total cost of the algorithms are different. I think you are confusing the maximum depth of the recursion with total running time of the algorithm.
The given function does a constant amount c of work before making two recursive function calls. Let c denote the work done by a function outside its recursive calls. You can draw a recursion tree where each node is the cost of a function call. The root node has a cost of c. The root node has two children because there are two recursive calls, each with a cost of c. Each of these children makes two further recursive calls; hence, the root node has 4 grandchildren, each with a cost of c. This continues until we hit the base case.
The total cost of the recursion tree is the cost of the root node (which is c), plus the cost of its children (which is 2c), plus the cost of the grandchilden (which is 4c), and so on, until we hit the n leaves (which have a total cost of nc, where for simplicity we'll assume n is a power of 2). The total cost of all levels of the recursion tree is c+2c+4c+8c+...+nc = O(nc) = O(n). Here, we used the fact that in an increasing geometric series, the total sum is dominated by the last term (the sum is essentially just the last term, up to constant factors, which are subsumed in asymptotic notation). This sum had O(log n) terms, but the sum is O(n).
Equivalently, the recurrence describing the running time of your algorithm is T(n) = 2T(n/2)+c, and by the Master theorem, the solution is T(n) = O(n). This is different from binary search, which has the recurrence T(n)=1T(n/2)+c, which has the solution T(n)=O(log n). For binary search, the total cost of all levels of the recursion tree would be c+c+...+c; here, the sum has O(log n) terms and the sum is O(log n).
I came across this question in one of the slides of Stanford, that what would be the effect on the complexity of the code of merge sort if we split the array into 4 or 8 instead of 2.
It would be the same: O(n log n). You will have a shorter tree and the base of the logarithm will change, but that doesn't matter for big-oh, because a logarithm in a base a differs from a logarithm in base b by a constant:
log_a(x) = log_b(x) / log_b(a)
1 / log_b(a) = constant
And big-oh ignores constants.
You will still have to do O(n) work per tree level in order to merge the 4 or 8 or however many parts, which, combined with more recursive calls, might just make the whole thing even slower in practice.
In general, you can split your array into equal size subarrays of any size and then sort the subarrays recursively, and then use a min-heap to keep extracting the next smallest element from the collection of sorted subarrays. If the number of subarrays you break into is constant, then the execution time for each min-heap per operation is constant, so you arrive at the same O(n log n) time.
Intuitively it would be the same as there is no much difference between splitting the array into two parts and then doing it again or splitting it to 4 parts from the beginning.
A more official proof by induction based on this (I'll assume that the array is split into k):
Definitions:
Let T(N) - number of array stores to mergesort of input of size N
Then mergesort recurrence T(N) = k*T(N/k) + N (for N > 1, T(1) = 0)
Claim:
If T(N) satisfies the recurrence above then T(N) = Nlg(N)
Note - all the logarithms below are on base k
Proof:
Base case: N=1
Inductive hypothesis: T(N) = NlgN
Goal: show that T(kN) = kN(lg(kN))
T(kN) = kT(N) + kN [mergesort recurrence]
= kNlgN + kN [inductive hypothesis]
= kN(lg[(kN/k)] [algebra]
= kN(lg(kN) - lgk) [algebra]
= kN(lg(kN) - 1) + kN [algebra - for base k, lg(k )= 1]
= kNlg(kN) [QED]
For 3-way Quicksort (dual-pivot quicksort), how would I go about finding the Big-O bound? Could anyone show me how to derive it?
There's a subtle difference between finding the complexity of an algorithm and proving it.
To find the complexity of this algorithm, you can do as amit said in the other answer: you know that in average, you split your problem of size n into three smaller problems of size n/3, so you will get, in è log_3(n)` steps in average, to problems of size 1. With experience, you will start getting the feeling of this approach and be able to deduce the complexity of algorithms just by thinking about them in terms of subproblems involved.
To prove that this algorithm runs in O(nlogn) in the average case, you use the Master Theorem. To use it, you have to write the recursion formula giving the time spent sorting your array. As we said, sorting an array of size n can be decomposed into sorting three arrays of sizes n/3 plus the time spent building them. This can be written as follows:
T(n) = 3T(n/3) + f(n)
Where T(n) is a function giving the resolution "time" for an input of size n (actually the number of elementary operations needed), and f(n) gives the "time" needed to split the problem into subproblems.
For 3-Way quicksort, f(n) = c*n because you go through the array, check where to place each item and eventually make a swap. This places us in Case 2 of the Master Theorem, which states that if f(n) = O(n^(log_b(a)) log^k(n)) for some k >= 0 (in our case k = 0) then
T(n) = O(n^(log_b(a)) log^(k+1)(n)))
As a = 3 and b = 3 (we get these from the recurrence relation, T(n) = aT(n/b)), this simplifies to
T(n) = O(n log n)
And that's a proof.
Well, the same prove actually holds.
Each iteration splits the array into 3 sublists, on average the size of these sublists is n/3 each.
Thus - number of iterations needed is log_3(n) because you need to find number of times you do (((n/3) /3) /3) ... until you get to one. This gives you the formula:
n/(3^i) = 1
Which is satisfied for i = log_3(n).
Each iteration is still going over all the input (but in a different sublist) - same as quicksort, which gives you O(n*log_3(n)).
Since log_3(n) = log(n)/log(3) = log(n) * CONSTANT, you get that the run time is O(nlogn) on average.
Note, even if you take a more pessimistic approach to calculate the size of the sublists, by taking minimum of uniform distribution - it will still get you first sublist of size 1/4, 2nd sublist of size 1/2, and last sublist of size 1/4 (minimum and maximum of uniform distribution), which will again decay to log_k(n) iterations (with a different k>2) - which will yield O(nlogn) overall - again.
Formally, the proof will be something like:
Each iteration takes at most c_1* n ops to run, for each n>N_1, for some constants c_1,N_1. (Definition of big O notation, and the claim that each iteration is O(n) excluding recursion. Convince yourself why this is true. Note that in here - "iteration" means all iterations done by the algorithm in a certain "level", and not in a single recursive invokation).
As seen above, you have log_3(n) = log(n)/log(3) iterations on average case (taking the optimistic version here, same principles for pessimistic can be used)
Now, we get that the running time T(n) of the algorithm is:
for each n > N_1:
T(n) <= c_1 * n * log(n)/log(3)
T(n) <= c_1 * nlogn
By definition of big O notation, it means T(n) is in O(nlogn) with M = c_1 and N = N_1.
QED
Can someone explain to me why this is true. I heard a professor mention this is his lecture
The two notions are orthogonal.
You can have worst case asymptotics. If f(n) denotes the worst case time taken by a given algorithm with input n, you can have eg. f(n) = O(n^3) or other asymptotic upper bounds of the worst case time complexity.
Likewise, you can have g(n) = O(n^2 log n) where g(n) is the average time taken by the same algorithm with (say) uniformly distributed (random) inputs of size n.
Or you can have h(n) = O(n) where h(n) is the average time taken by the same algorithm with particularly distributed random inputs of size n (eg. almost sorted sequences for a sorting algorithm).
Asymptotic notation is a "measure". You have to specify what you want to count: worst case, best case, average, etc.
Sometimes, you are interested in stating asymptotic lower bounds of (say) the worst case complexity. Then you write f(n) = Omega(n^2) to state that in the worst case, the complexity is at least n^2. The big-Omega notation is opposite to big-O: f = Omega(g) if and only if g = O(f).
Take quicksort for an example. Each successive recursive call n of quicksort has a run-time complexity T(n) of
T(n) = O(n) + 2 T[ (n-1)/2 ]
in the 'best case' if the unsorted input list is splitted into two equal sublists of size (n-1)/2 in each call. Solving for T(n) gives O(n log n), in this case. If the partition is not perfect, and the two sublists are not of equal size n, i.e.
T(n) = O(n) + T(k) + T(n - 1 - k),
we still obtain O(n log n) even if k=1, just with a larger constant factor. This is because the number of recursive calls of quicksort is rising exponentially while processing the input list as long as k>0.
However, in the 'worst case' no division of the input list takes place, i.e.:
T(n) = O(n) + T(0) + T(n - 1) = O(n) + O(n-1) + T(n-1) + T(n-2) ... .
This happens e.g. if we take the first element of a sorted list as the pivot element.
Here, T(0) means one of the resulting sublists is zero and therefore takes no computing time (since the sublist has zero elements). All the remaining load T(n-1) is needed for the second sublist. In this case, we obtain O(n²).
If an algorithm had no worst case scenario, it would be not only be O[f(n)] but also o[f(n)] (Big-O vs. little-o notation).
The asymptotic bound is the expected behaviour as the number of operations go to infinity. Mathematically it is just that lim as n goes to infinity. The worst case behaviour however is applicable to finite number of operations.
What's the complexity of a recursive program to find factorial of a number n? My hunch is that it might be O(n).
If you take multiplication as O(1), then yes, O(N) is correct. However, note that multiplying two numbers of arbitrary length x is not O(1) on finite hardware -- as x tends to infinity, the time needed for multiplication grows (e.g. if you use Karatsuba multiplication, it's O(x ** 1.585)).
You can theoretically do better for sufficiently huge numbers with Schönhage-Strassen, but I confess I have no real world experience with that one. x, the "length" or "number of digits" (in whatever base, doesn't matter for big-O anyway of N, grows with O(log N), of course.
If you mean to limit your question to factorials of numbers short enough to be multiplied in O(1), then there's no way N can "tend to infinity" and therefore big-O notation is inappropriate.
Assuming you're talking about the most naive factorial algorithm ever:
factorial (n):
if (n = 0) then return 1
otherwise return n * factorial(n-1)
Yes, the algorithm is linear, running in O(n) time. This is the case because it executes once every time it decrements the value n, and it decrements the value n until it reaches 0, meaning the function is called recursively n times. This is assuming, of course, that both decrementation and multiplication are constant operations.
Of course, if you implement factorial some other way (for example, using addition recursively instead of multiplication), you can end up with a much more time-complex algorithm. I wouldn't advise using such an algorithm, though.
When you express the complexity of an algorithm, it is always as a function of the input size. It is only valid to assume that multiplication is an O(1) operation if the numbers that you are multiplying are of fixed size. For example, if you wanted to determine the complexity of an algorithm that computes matrix products, you might assume that the individual components of the matrices were of fixed size. Then it would be valid to assume that multiplication of two individual matrix components was O(1), and you would compute the complexity according to the number of entries in each matrix.
However, when you want to figure out the complexity of an algorithm to compute N! you have to assume that N can be arbitrarily large, so it is not valid to assume that multiplication is an O(1) operation.
If you want to multiply an n-bit number with an m-bit number the naive algorithm (the kind you do by hand) takes time O(mn), but there are faster algorithms.
If you want to analyze the complexity of the easy algorithm for computing N!
factorial(N)
f=1
for i = 2 to N
f=f*i
return f
then at the k-th step in the for loop, you are multiplying (k-1)! by k. The number of bits used to represent (k-1)! is O(k log k) and the number of bits used to represent k is O(log k). So the time required to multiply (k-1)! and k is O(k (log k)^2) (assuming you use the naive multiplication algorithm). Then the total amount of time taken by the algorithm is the sum of the time taken at each step:
sum k = 1 to N [k (log k)^2] <= (log N)^2 * (sum k = 1 to N [k]) =
O(N^2 (log N)^2)
You could improve this performance by using a faster multiplication algorithm, like Schönhage-Strassen which takes time O(n*log(n)*log(log(n))) for 2 n-bit numbers.
The other way to improve performance is to use a better algorithm to compute N!. The fastest one that I know of first computes the prime factorization of N! and then multiplies all the prime factors.
The time-complexity of recursive factorial would be:
factorial (n) {
if (n = 0)
return 1
else
return n * factorial(n-1)
}
So,
The time complexity for one recursive call would be:
T(n) = T(n-1) + 3 (3 is for As we have to do three constant operations like
multiplication,subtraction and checking the value of n in each recursive
call)
= T(n-2) + 6 (Second recursive call)
= T(n-3) + 9 (Third recursive call)
.
.
.
.
= T(n-k) + 3k
till, k = n
Then,
= T(n-n) + 3n
= T(0) + 3n
= 1 + 3n
To represent in Big-Oh notation,
T(N) is directly proportional to n,
Therefore,
The time complexity of recursive factorial is O(n).
As there is no extra space taken during the recursive calls,the space complexity is O(N).