I would like to come up with a recurrence for this given problem:
Consider a variation of the randomized quicksort algorithm where the pivot is picked randomly until the array is partitioned in such a way that both the lower subarray L and the greater subarray G
contain 3/4 of the elements of the array. For instance, if the randomly chosen pivot
partitions the array in such a way that L contains 1/10 of the elements, then another
pivot is randomly chosen. Analyze the expected running time of this algorithm.
At first I treated this question as if it's just a regular quicksort question and came up with this recurrence where:
T(n) = T(3/4n) + T(n/4) + Θ(n) (where Θ(n) comes from the partition)
It would make sense if we had an algorithm where the split is always 1/4 : 3/4. But we are using random pivotting here and pivot changes everytime the condition for partioning is not satisfied. I know that worst-case running time for randomized quicksort is still O(n^2) but I think under these circumstances the worst case is different now (something worse than O(n^2)). Am I on the right track so far?
The time complexity of quick sort will never go beyond O(n^2) unless you chose some logic which takes O(n) time to chose the pivot.
The best way to chose the pivot is a random element or end or first element.
There are n/2 bad pivots. Assuming you never select the same pivot twice (if you do, the worst case is always selecting a bad pivot, i.e. infinite time), in the worst case you'd repeat the partitioning n/2 times, which leads to Θ(n^2) complexity of partitioning phase. The recurrence becomes
T(n) = T(n/4) + T(3n/4) + Θ(n^2)
Time complexity of Quicksort when pivot always is the 2nd smallest element in a sublist.
Is it still O(NlogN)?
If i solve the recurrence equation
F(N) = F(N-2) + N
= F(N-2(2)) + 2N -2
= F(N-3(2)) + 3N - (2+1)(2)
= F(N-4(2)) + 4N - (3+2+1)(2)
Which is O(N^2), but I doubt my answer somehow, someone help me with the clarification please?
To start with, the quicksort algorithm has an average time complexity of O(NlogN), but its worst-time complexity is actually O(N^2).
The generic complexity analysis of quicksort depends not just on the devising of the recurrence relations, but also on the value of the variable K in F(N-K) term of your recurrence relation. And according to whether you're calculating best, average and worst case complexities, that value is usually estimated by the probability distribution of having the best, average, or worst element as the pivot, respectively.
If, for instance, you want to compute the best case, then you may think that your pivot always divides the array into two. (i.e. K=N/2) If computing for the worst case, you may think that your pivot is either the largest or the smallest element. (i.e. K=1) For the average case, based on the probability distribution of the indices of the elements, K=N/4 is used. (You may need more about it here) Basically, for the average case, your recurrence relation becomes F(N) = F(N / 4) + F(3 * N / 4) + N, which yields O(NlogN).
Now, the value you assumed for K, namely 2, is just one shy from the worst case scenario. That is why you can not observe the average case performance of O(NlogN) here, and get O(N^2).
Quick is the in place algorithm which does not use any auxiliary array. So why memory complexity of this O(nlog(n)) ?
Similarly I understand it's worst case time complexity is O(n^2) but not getting why average case time complexity is O(nlog(n)). Basically I am not sure what do we mean when we say average case complexity ?
To your second point an excerpt from Wikipedia:
The most unbalanced partition occurs when the partitioning routine returns one of sublists of size n − 1. This may occur if the pivot happens to be the smallest or largest element in the list, or in some implementations (e.g., the Lomuto partition scheme as described above) when all the elements are equal.
If this happens repeatedly in every partition, then each recursive call processes a list of size one less than the previous list. Consequently, we can make n − 1 nested calls before we reach a list of size 1. This means that the call tree is a linear chain of n − 1 nested calls. The ith call does O(n − i) work to do the partition, and {\displaystyle \textstyle \sum _{i=0}^{n}(n-i)=O(n^{2})} , so in that case Quicksort takes O(n²) time.
Because you usually don't know what exact numbers you have to sort and you don't know, which pivot element you choose, you have the chance, that your pivot element isn't the smallest or biggest number in the array you sort. If you have an array of n not duplicated numbers, you have the chance of (n - 2) / n, that you don't have a worst case.
Given the list of numbers:
2 5 1 8 4 10 6 3 7 9 0
The actual implementation of quick sort I understand, but a question on my homework that I didn't was:
What is the optimal choice of pivot, why?
I had assumed when reading this that the obvious choice for a pivot would be the 5 or 6 since its in the middle of the list. I figured quick sort would work either way though since we choose a new pivot every time. Which makes the followup question make a little more sense, but does anyone have a formal definition?
Why is an optimal pivot not practical?
The optimal pivot is the median of the set you're currently working on, because it will split the set into two equal-sized subsets which guarantees O(n log n) performance. The reason it's not practical is because of the cost of finding the actual median. You essentially have to sort the data to find the median, so it's like the book Catch 22 - "How do I sort the data?" "Find the median" "How do I find a median?" "Sort the data".
Optimal pivot is in the middle, because when you move it to the left or to the right (or take biggest or smallest item), you increase depth of recursion. In the worst case you will get O(n^2) except of O(n*log2(n)) when taking the middle.
Optimal pivot must be median of numbers because then subproblem sizes are exactly half of original. The time complexity is defined as follows:-
T(N) = T(N/2) + O(N)
which evaluates to
T(N) = O(NlogN)
Whereas if pivot ends up being the first element of array after partitioning then:-
T(N) = T(N-1) + O(N)
T(N) = O(N^2)
which is as bad as bubble sort
The reason that using median always as pivot is not practical because the algorithm that do it in O(N) are very complex & u can always do it in O(NlogN) but that is sorting again which is the problem which we are solving. Here is an example of algorithm that evaluates median in O(N) : -
Median of Medians
I know there are quite a bunch of questions about big O notation, I have already checked:
Plain english explanation of Big O
Big O, how do you calculate/approximate it?
Big O Notation Homework--Code Fragment Algorithm Analysis?
to name a few.
I know by "intuition" how to calculate it for n, n^2, n! and so, however I am completely lost on how to calculate it for algorithms that are log n , n log n, n log log n and so.
What I mean is, I know that Quick Sort is n log n (on average).. but, why? Same thing for merge/comb, etc.
Could anybody explain me in a not too math-y way how do you calculate this?
The main reason is that Im about to have a big interview and I'm pretty sure they'll ask for this kind of stuff. I have researched for a few days now, and everybody seem to have either an explanation of why bubble sort is n^2 or the unreadable explanation (for me) on Wikipedia
The logarithm is the inverse operation of exponentiation. An example of exponentiation is when you double the number of items at each step. Thus, a logarithmic algorithm often halves the number of items at each step. For example, binary search falls into this category.
Many algorithms require a logarithmic number of big steps, but each big step requires O(n) units of work. Mergesort falls into this category.
Usually you can identify these kinds of problems by visualizing them as a balanced binary tree. For example, here's merge sort:
6 2 0 4 1 3 7 5
2 6 0 4 1 3 5 7
0 2 4 6 1 3 5 7
0 1 2 3 4 5 6 7
At the top is the input, as leaves of the tree. The algorithm creates a new node by sorting the two nodes above it. We know the height of a balanced binary tree is O(log n) so there are O(log n) big steps. However, creating each new row takes O(n) work. O(log n) big steps of O(n) work each means that mergesort is O(n log n) overall.
Generally, O(log n) algorithms look like the function below. They get to discard half of the data at each step.
def function(data, n):
if n <= constant:
return do_simple_case(data, n)
if some_condition():
function(data[:n/2], n / 2) # Recurse on first half of data
else:
function(data[n/2:], n - n / 2) # Recurse on second half of data
While O(n log n) algorithms look like the function below. They also split the data in half, but they need to consider both halves.
def function(data, n):
if n <= constant:
return do_simple_case(data, n)
part1 = function(data[n/2:], n / 2) # Recurse on first half of data
part2 = function(data[:n/2], n - n / 2) # Recurse on second half of data
return combine(part1, part2)
Where do_simple_case() takes O(1) time and combine() takes no more than O(n) time.
The algorithms don't need to split the data exactly in half. They could split it into one-third and two-thirds, and that would be fine. For average-case performance, splitting it in half on average is sufficient (like QuickSort). As long as the recursion is done on pieces of (n/something) and (n - n/something), it's okay. If it's breaking it into (k) and (n-k) then the height of the tree will be O(n) and not O(log n).
You can usually claim log n for algorithms where it halves the space/time each time it runs. A good example of this is any binary algorithm (e.g., binary search). You pick either left or right, which then axes the space you're searching in half. The pattern of repeatedly doing half is log n.
For some algorithms, getting a tight bound for the running time through intuition is close to impossible (I don't think I'll ever be able to intuit a O(n log log n) running time, for instance, and I doubt anyone will ever expect you to). If you can get your hands on the CLRS Introduction to Algorithms text, you'll find a pretty thorough treatment of asymptotic notation which is appropriately rigorous without being completely opaque.
If the algorithm is recursive, one simple way to derive a bound is to write out a recurrence and then set out to solve it, either iteratively or using the Master Theorem or some other way. For instance, if you're not looking to be super rigorous about it, the easiest way to get QuickSort's running time is through the Master Theorem -- QuickSort entails partitioning the array into two relatively equal subarrays (it should be fairly intuitive to see that this is O(n)), and then calling QuickSort recursively on those two subarrays. Then if we let T(n) denote the running time, we have T(n) = 2T(n/2) + O(n), which by the Master Method is O(n log n).
Check out the "phone book" example given here: What is a plain English explanation of "Big O" notation?
Remember that Big-O is all about scale: how much more operation will this algorithm require as the data set grows?
O(log n) generally means you can cut the dataset in half with each iteration (e.g. binary search)
O(n log n) means you're performing an O(log n) operation for each item in your dataset
I'm pretty sure 'O(n log log n)' doesn't make any sense. Or if it does, it simplifies down to O(n log n).
I'll attempt to do an intuitive analysis of why Mergesort is n log n and if you can give me an example of an n log log n algorithm, I can work through it as well.
Mergesort is a sorting example that works through splitting a list of elements repeatedly until only elements exists and then merging these lists together. The primary operation in each of these merges is comparison and each merge requires at most n comparisons where n is the length of the two lists combined. From this you can derive the recurrence and easily solve it, but we'll avoid that method.
Instead consider how Mergesort is going to behave, we're going to take a list and split it, then take those halves and split it again, until we have n partitions of length 1. I hope that it's easy to see that this recursion will only go log (n) deep until we have split the list up into our n partitions.
Now that we have that each of these n partitions will need to be merged, then once those are merged the next level will need to be merged, until we have a list of length n again. Refer to wikipedia's graphic for a simple example of this process http://en.wikipedia.org/wiki/File:Merge_sort_algorithm_diagram.svg.
Now consider the amount of time that this process will take, we're going to have log (n) levels and at each level we will have to merge all of the lists. As it turns out each level will take n time to merge, because we'll be merging a total of n elements each time. Then you can fairly easily see that it will take n log (n) time to sort an array with mergesort if you take the comparison operation to be the most important operation.
If anything is unclear or I skipped somewhere please let me know and I can try to be more verbose.
Edit Second Explanation:
Let me think if I can explain this better.
The problem is broken into a bunch of smaller lists and then the smaller lists are sorted and merged until you return to the original list which is now sorted.
When you break up the problems you have several different levels of size first you'll have two lists of size: n/2, n/2 then at the next level you'll have four lists of size: n/4, n/4, n/4, n/4 at the next level you'll have n/8, n/8 ,n/8 ,n/8, n/8, n/8 ,n/8 ,n/8 this continues until n/2^k is equal to 1 (each subdivision is the length divided by a power of 2, not all lengths will be divisible by four so it won't be quite this pretty). This is repeated division by two and can continue at most log_2(n) times, because 2^(log_2(n) )=n, so any more division by 2 would yield a list of size zero.
Now the important thing to note is that at every level we have n elements so for each level the merge will take n time, because merge is a linear operation. If there are log(n) levels of the recursion then we will perform this linear operation log(n) times, therefore our running time will be n log(n).
Sorry if that isn't helpful either.
When applying a divide-and-conquer algorithm where you partition the problem into sub-problems until it is so simple that it is trivial, if the partitioning goes well, the size of each sub-problem is n/2 or thereabout. This is often the origin of the log(n) that crops up in big-O complexity: O(log(n)) is the number of recursive calls needed when partitioning goes well.