In Cormen's own words - "The difference is that with the deterministic algorithm, a particular input can elicit that worst-case behavior. With the randomized algorithm, however, no input can always elicit the worst-case behavior."
How does adding a randomized pivot change anything, the algorithm is still gonna perform bad under some particular input and considering each kind of input equally likely this is no better than that of the standard quicksort, only difference being we don't actually know which particular input is going to cause the worst case time complexity. So why is the randomized version considered better?
Consider the following version of quicksort, where we always pick the last element as the pivot. Now consider the following array:
int[] arr = {9, 8, 7, 6, 5, 4, 3, 2, 1};
When this array is sorted using our version of quicksort, it will always pick the smallest element as its pivot, the last element. And in the first iteration, it will change the array like this:
arr = [1, 8, 7, 6, 5, 4, 3, 2, 9];
Now, it will recurse on the sub-arrays:
s1 = [1, 8, 7, 6, 5, 4, 3, 2];
s2 = [9];
In s1 it will again pick 2 as its pivot, and only 8 and 2 will interchange positions. So, in this way, if we try to formulate a recurrence relation, for its complexity, it will be
T(n) = T(n-1) + O(n)
which corresponds to O(n^2).
So, for this array, the standard version will always take O(n^2) time.
In the randomized version, we first exchange the last element with some random element in the array and then select it as the pivot. So, for the given array, this pivot will split the array randomly, most probably in the middle. So, now the recurrence will be
T(n) = 2T(n/2) + O(n)
which will be O(n * Log(n)).
That's why we consider randomized quicksort better than standard quicksort, because, there is very low probability of bad splits in randomized quicksort.
The difference is that with the deterministic algorithm, a particular input can elicit that worst-case behavior. With the randomized algorithm, however, no input can always elicit the worst-case behavior.
This should be clarified to mean a truly randomized algorithm. If instead a deterministic pseudo-random algorithm is used, then a deliberately created input can elicit worst case behavior.
With the randomized algorithm, however, no input can always elicit the worst-case behavior.
This should be clarified: even with a truly randomized algorithm, there is still the possibility of some specific input that could elicit worst-case behavior in one or more invocations of a randomized quicksort with that input, but no input could always elicit worst-case behavior for an infinite number of invocations of a truly randomized quicksort on that same input.
Most library implementations of single pivot quicksort use a median of 3 or median of 9, since they can't rely on having fast instructions for random numbers like X86 RRAND and fast divide (for modulo function). If a quicksort was somehow part of an encryption scheme, then a truly randomized algorithm could be used to avoid time based attacks.
Related
Encountered a question like this for mergesort specifically and was wondering how does one approach a question like this for other algorithms (insertionsort,heapsort,quicksort and etc)
Is it safe to assume that the nth best/worst arrangement for any algorithm is the nth step of solving the best/worst arrangement for the same set of data?
Example:
If the worst case for mergesort with the following array of integers [1,2,3,4,5,6,7,8] is [1,5,3,7,2,6,4,8]. What is the next worst case for this array of integers?
I assumed it would be the next arrangement when solving the worst case which is [1,3,5,7,2,6,4,8]. Am I approaching such a question wrongly?
The concept of a "next-best" or "next-worst" case is not really well-defined in the first place. Neither is the concept of "the state of the array after one step", because not all algorithms modify an array in-place.
When we say the "worst case" of an algorithm, we don't mean a single input to an algorithm. For example, the array [5, 4, 3, 2, 1] is not - by itself - the worst case of the insertion sort algorithm. This array is one of the worst inputs (i.e. highest number of steps to compute) for insertion sort out of arrays of length 5, but we are very rarely interested in arrays of one specific length.
What we mean by "best case" or "worst case" is actually an infinite family of inputs, such that each member of that family is a best or worst input for its own value of n, and the family must contain inputs for arbitrarily large values of n. So, for example:
The infinite set of arrays {[1], [2, 1], [3, 2, 1], [4, 3, 2, 1], ...} is a worst case for insertion sort. For inputs from this infinite set, the asymptotic complexity of insertion sort is Θ(n2).
The infinite set of arrays {[1], [1, 2], [1, 2, 3], [1, 2, 3, 4], ...} is a best case for insertion sort. For inputs from this infinite set, the asymptotic complexity of insertion sort is Θ(n).
Note that the (larger) infinite set of all arrays which are in descending order is also a worst case for insertion sort, and the (larger) infinite set of all arrays in ascending order is also a best case. So the family is not unique, but the asymptotic complexity of the algorithm on inputs from any two "best case" (or any two "worst case") families is the same.
Now we've got that out of the way, let's think about what a "next-best" or "next-worst" case would have to mean. If the asymptotic complexity of insertion sort on some family of inputs is also Θ(n2), then that family is a worst case for insertion sort; so the asymptotic complexity of a "next-worst" case would have to be something lower than Θ(n2).
But however small a gap you choose, it is not the "next-worst":
If you choose a family where the complexity is Θ(n1.999), then it is not "next-worst" because I can find another family where the complexity is Θ(n1.9999).
If you choose a family where the complexity is Θ(n2 / log n), I can find one where it's Θ(n2 / log log n).
That is, the asymptotic complexities of different families of possible inputs for insertion sort form a dense order, for any two different complexities there is another complexity in between those two, so there is no "next" or "previous" one.
In the Subset Sum problem, if we don't use the Dynamic Programming approach, then we have an exponential time complexity. But if we draw the recursion tree, it seems that all the 2^n branches are unique. If we use dynamic programming, how can we assure that all the unique branches are explored? If there really exists 2^n possible solutions, how does dynamic programming reduce it to polynomial time while also ensuring all 2^n solutions are explored?
How does dynamic programming reduce it to polynomial time while also ensuring all 2^n solutions are explored?
It is pseudo polynomial time, not polynomial time. It's a very important distinction. According to Wikipedia, A numeric algorithm runs in pseudo-polynomial time if its running time is a polynomial in the numeric value of the input, but not necessarily in the length of the input, which is the case for polynomial time algorithms.
What does it matter?
Consider an example [1, 2, 3, 4], sum = 1 + 2 + 3 + 4 = 10.
There does in fact exist 2^4 = 16 subsequences, however, do we need to check them all? The answer is no, since we are only concerned about the sum of subsequence. To illustrate this, let's say we're iterating from the 1st element to the 4th element:
1st element:
We can choose to take or not take the 1st element, so the possible sum will be [0, 1].
2nd element:
We can choose to take or not to take the 2nd element. Same idea, possible sum will be [0, 1, 2, 3].
3rd element:
We have [0, 1, 2, 3] now. We now consider taking the third element. But wait... If we take the third element and add it to 0, we still get 3, which is already present in the array, do we need to store this piece of information? Apparently not. In fact, we only need to know whether a sum is possible at any stage. If there are multiple subsequences summing to the same value, we ignore it. This is the key to the reduction of complexity, if you consider it as a reduction.
With that said, a real polynomial solution for subset sum is not known since it is NP-complete
I have found out that selection sort uses Brute Force strategy. However, I think that it uses Greedy strategy.
Why do I think that it uses Greedy: it goes from 0 to n-1 at it outer loop and from i+1 to n-1. This is really naive. It selects the minimum element in one every iteration - it chooses best locally. Everything like in Greedy but it is not.
Can you please explain me why it is not how I think? Information about this issue I have not found in the Internet.
A selection sort could indeed be described as a greedy algorithm, in the sense that it:
tries to choose an output (a permutation of its inputs) that optimizes a certain measure ("sortedness", which could be measured in various ways, e.g. by number of inversions), and
does so by breaking the task into smaller subproblems (for selection sort, finding the k-th element in the output permutation) and picking the locally optimal solution to each subproblem.
As it happens, the same description could be applied to most other sorting algorithms, as well — the only real difference is the choice of subproblems. For example:
insertion sort locally optimizes the sortedness of the permutation of k first input elements;
bubble sort optimizes the sortedness of adjacent pairs of elements; it needs to iterate over the list several times to reach a global optimum, but this still falls within the broad definition of a greedy algorithm;
merge sort optimizes the sortedness of exponentially growing subsequences of the input sequence;
quicksort recursively divides its input into subsequences on either side of an arbitrarily chosen pivot, optimizing the division to maximize sortedness at each stage.
Indeed, off the top of my head, I can't think of any practical sorting algorithm that wouldn't be greedy in this sense. (Bogosort isn't, but can hardly be called practical.) Furthermore, formulating these sorting algorithms as greedy optimization problems like this rather obscures the details that actually matter in practice when comparing sorting algorithms.
Thus, I'd say that characterizing selection sort, or any other sorting algorithm, as greedy is technically valid but practically useless, since such classification provides no real useful information.
Let A be a list of intgers such that: A = [5, 4, 3, 6, 1, 2, 7 ]
A greedy algorithm will look for the most promising direction, therefore :
we will compare: 5 to 4, see that 4 is indeed smaller than 5, set 4 as our minimum
compare 4 to 3 , set 3 as our minimum
Now we compare 3 to 6 and here is the tricky part: while in a normal selection sort(brute force) we will keep considering the remaining numbers, in a greedy approach we will take 3 as our minimum and will not consider the remaining numbers, hence "Best locally".
so a sorted list using this approach will result in a list sorted as such:
[3, 4, 5, 1, 2, 7]
Greedy and brute force describe different traits of the algorithm.
Greedy means that the algorithm on each step selects some option which is locally the best. That is, it have no look-ahead.
Brute-force means that the algorithm looks for an options in a straightforward manner, considering them all. E.g. it might search for an element via binary search, and it wouldn't be brute force anymore.
So the algorithm may be both greedy and brute force. These qualities are not mutually exclusive.
How is it different if I select a randomized pivot versus just selecting the first pivot in an unordered set/list?
If the set is unordered, isnt selecting the first value in the set, random in itself? So essentially, I am trying to understand how/if randomizing promises a better worst case runtime.
I think you may be mixing up the concepts of arbitrary and random. It's arbitrary to pick the first element of the array - you could pick any element you'd like and it would work equally well - but it's not random. A random choice is one that can't be predicted in advance. An arbitrary choice is one that can be.
Let's imagine that you're using quicksort on the sorted sequence 1, 2, 3, 4, 5, 6, ..., n. If you choose the first element as a pivot, then you'll choose 1 as the pivot. All n - 1 other elements then go to the right and nothing goes to the left, and you'll recursively quicksort 2, 3, 4, 5, ..., n.
When you quicksort that range, you'll choose 2 as the pivot. Partitioning the elements then puts nothing on the left and the numbers 3, 4, 5, 6, ..., n on the right, so you'll recursively quicksort 3, 4, 5, 6, ..., n.
More generally, after k steps, you'll choose the number k as a pivot, put the numbers k+1, k+2, ..., n on the right, then recursively quicksort them.
The total work done here ends up being Θ(n2), since on the first pass (to partition 2, 3, ..., n around 1) you have to look at n-1 elements, on the second pass (to partition 3, 4, 5, ..., n around 2), you have to look at n-2 elements, etc. This means that the work done is (n-1)+(n-2)+ ... +1 = Θ(n2), quite inefficient!
Now, contrast this with randomized quicksort. In randomized quicksort, you truly choose a random element as your pivot at each step. This means that while you technically could choose the same pivots as in the deterministic case, it's very unlikely (the probability would be roughly 22 - n, which is quite low) that this will happen and trigger the worst-case behavior. You're more likely to choose pivots closer to the center of the array, and when that happens the recursion branches more evenly and thus terminates a lot faster.
The advantage of randomized quicksort is that there's no one input that will always cause it to run in time Θ(n log n) and the runtime is expected to be O(n log n). Deterministic quicksort algorithms usually have the drawback that either (1) they run in worst-case time O(n log n), but with a high constant factor, or (2) they run in worst-case time O(n2) and the sort of input that triggers this case is deterministic.
In quick sort, the pivot is always the right most index of the selected array whereas in Randomized quick sort, pivot can be any element in the array.
Using an algorithm Tree-Insert(T, v) that inserts a new value v into a binary search tree T, the following algorithm grows a binary search tree by repeatedly inserting each value in a given section of an array into the tree:
Tree-Grow(A, first, last, T)
1 for i ← first to last
2 do Tree-Insert(T, A[i])
If the tree is initially empty, and the length of array section (i.e., last-first+1) is n, what are the best-case and the worst-case asymptotic running time of the above algorithm, respectively?
When n = 7, give a best-case instance (as an array containing digits 1 to 7, in certain order), and a worst-case instance (in the same form) of the algorithm.
If the array is sorted and all the values are distinct, find a way to modify Tree-Grow, so that it will always build the shortest tree.
What are the best-case and the worst-case asymptotic running time of the modified algorithm, respectively?
Please tag homework questions with the homework tag. In order to do well on your final exam, I suggest you actually learn this stuff, but I'm not here to judge you.
1) It takes O(n) to iterate from first to last. It takes O(lg n) to insert into a binary tree, therefore it the algorithm that you have shown takes O(n lg n) in the best case.
The worst case of inserting into a binary tree is when the tree is really long, but not very bushy; similar to a linked list. In that case, it would take O(n) to insert, therefore it would take O(n^2) in the worst case.
2) Best Case: [4, 2, 6, 1, 3, 5, 7], Worst Case: [1, 2, 3, 4, 5, 6, 7]
3) Use the n/2 index as the root, then recursively do this for the left side and right side of the array.
4) O(n lg n) in the best and worst case.
I hope this helps.