As given in books radix sort is generally done from LSB to MSB . But i think by using MSB to LSB we can divide input elements into smaller partitions.. Now I have to construct an example to show that MSB first radix sort can be asymptotically worse than LSB first radix sort.. This is my assignment problem.
Let the range of sorting be [1, n2], i.e. we need two applications of bucket sorting in the range [1, n]. If we use MSB first sorting, then the time bound is ∑(i=1 to n) O(ni +n), where ni is the number of elements in the bucket (of the MSB). The range is [1, n] for the LSB.
Also sorting m numbers in the range [1, n] takes time O(m + n).
The value of ∑(i=1 to n) O(ni +n) = n + ∑(i=1 to n) n can be Ω(n2).
Radix sort is not as limited as first appearances might suggest. It's now the heart of a lot of GPGPU sorts (hurrah! bitonic sort goes back in the closet). I wrote up the basic way to make radix sort practical for many datasets where the keys can be ordered by memcmp(3) in this google doc on radix sort --- copying the diagrams into stackoverflow text boxes is beyond me :-) The one big "hurdle" is to not follow the "divide and conquer" approach of quicksort, since it repeats the expensive start-up cost of MSB radix sort. The radix sort equivalent of "divide and conquer" is to reduce the input set into a list of sorted and unsorted ranges, and handles all remaining unsorted ranges together in each pass. BTW you can see an application for bitonic sort in sorting doubles in SSE2 registers. HTH
Related
Consider two k-bit numbers (in binary representation):
$$A = A_1 A_2 A_3 A_4 ... A_k $$
$$B = B_1 B_2 B_3 B_4 ... B_k $$
to compare we scan from left to right looking for an occurrence of a 0 and check opposite number if that digit is also a 0 (for both numbers) noticing that if ever such a case is found then the source of the 0 is less than the source of the 1. But what if the numbers are:
111111111111
111111111110
clearly this will require scanning the whole number and if we are told nothing about the numbers ahead of time and simply given them then:
Comparison take $O(k)$ time.
Therefore when we look at the code for a sorting method such as high-performance quick sort:
HPQuicksort(list): T(n)
check if list is sorted: if so return list
compute median: O(n) time (or technically: O(nk))
Create empty list $L_1$, $L_2$, and $L_3$ O(1) time
Scan through list O(n)
if element is less place into $L_1$ O(k)
if element is more place into $L_2$ O(k)
if element is equal place into $L_3$ O(k)
return concatenation of HP sorted $L_1$, $L_3$, $L_2$ 2 T(n/2)
Thus: T(n) = O(n) + O(nk) + 2*T(n/2) ---> T(n) = O(nklog(n))
Which means quicksort is slower than radix sort.
Why do we still use it then?
There seem to be two independent questions here:
Why do we claim that comparisons take time O(1) when analyzing sorting algorithms, when in reality they might not?
Why would we use quicksort on large integers instead of radix sort?
For (1), typically, the runtime analysis of sorting algorithms is measured in terms of the number of comparisons made rather than in terms of the total number of operations performed. For example, the famous sorting lower bound gives a lower bound in terms of number of comparisons, and the analyses of quicksort, heapsort, selection sort, etc. all work by counting comparisons. This is useful for a few reasons. First, typically, a sorting algorithm will be implemented by being given an array and some comparison function used to compare them (for example, C's qsort or Java's Arrays.sort). From the perspective of the sorting algorithm, this is a black box. Therefore, it makes sense to analyze the algorithm by trying to minimize the number of calls to the black box. Second, if we do perform our analyses of sorting algorithms by counting comparisons, it's easy to then determine the overall runtime by multiplying the number of comparisons by the cost of a comparison. For example, you correctly determined that sorting n k-bit integers will take expected time O(kn log n) using quicksort, since you can just multiply the number of comparisons by the cost of a comparison.
For your second question - why would we use quicksort on large integers instead of radix sort? - typically, you would actually use radix sort in this context, not quicksort, for the specific reason that you pointed out. Quicksort is a great sorting algorithm for sorting objects that can be compared to one another and has excellent performance, but radix sort frequently outperforms it on large arrays of large strings or integers.
Hope this helps!
I know that counting and radix sorts are generally considered to run in O(n) time, and I believe I understand why. I'm being asked in an assignment, however, to explain why these sorts may not necessarily sort a list of distinct, positive integers in O(n) time. I can't come up with any reason.
Any help is appreciated!
To say that counting or radix sort is O(n) is actually not correct.
Counting sort is O(n+k) where k is the maximum value of any element in the array.
The reason is the you have to step through the entire list to populate the counts (O(n)), then step through the counts (O(k)).
Radix sort is O(mn) where m is the maximum number of digits of a number.
The reason is that you have to step through the array once for each digit (O(n) m times).
Consider the following input :
8,4,15,9,32,44,55
Propose an algorithm to sort these in ascending order in less than n*n time complexity.
Without using any divide and conquer approach
For relatively small integers such as these you can use bucket sort or radix sort, it will be O(n), and is not devide and conquer.
For larger ints it is still possible, but bucket sort will consume too much space, and radix sort will be O(n*d), where d is the number of digits in the biggest number.
Is it theoretically possible to sort an array of n integers in an amortized complexity of O(n)?
What about trying to create a worst case of O(n) complexity?
Most of the algorithms today are built on O(nlogn) average + O(n^2) worst case.
Some, while using more memory are O(nlogn) worst.
Can you with no limitation on memory usage create such an algorithm?
What if your memory is limited? how will this hurt your algorithm?
Any page on the intertubes that deals with comparison-based sorts will tell you that you cannot sort faster than O(n lg n) with comparison sorts. That is, if your sorting algorithm decides the order by comparing 2 elements against each other, you cannot do better than that. Examples include quicksort, bubblesort, mergesort.
Some algorithms, like count sort or bucket sort or radix sort do not use comparisons. Instead, they rely on the properties of the data itself, like the range of values in the data or the size of the data value.
Those algorithms might have faster complexities. Here is an example scenario:
You are sorting 10^6 integers, and each integer is between 0 and 10. Then you can just count the number of zeros, ones, twos, etc. and spit them back out in sorted order. That is how countsort works, in O(n + m) where m is the number of values your datum can take (in this case, m=11).
Another:
You are sorting 10^6 binary strings that are all at most 5 characters in length. You can use the radix sort for that: first split them into 2 buckets depending on their first character, then radix-sort them for the second character, third, fourth and fifth. As long as each step is a stable sort, you should end up with a perfectly sorted list in O(nm), where m is the number of digits or bits in your datum (in this case, m=5).
But in the general case, you cannot sort faster than O(n lg n) reliably (using a comparison sort).
I'm not quite happy with the accepted answer so far. So I'm retrying an answer:
Is it theoretically possible to sort an array of n integers in an amortized complexity of O(n)?
The answer to this question depends on the machine that would execute the sorting algorithm. If you have a random access machine, which can operate on exactly 1 bit, you can do radix sort for integers with at most k bits, which was already suggested. So you end up with complexity O(kn).
But if you are operating on a fixed size word machine with a word size of at least k bits (which all consumer computers are), the best you can achieve is O(n log n). This is because either log n < k or you could do a count sort first and then sort with a O (n log n) algorithm, which would yield the first case also.
What about trying to create a worst case of O(n) complexity?
That is not possible. A link was already given. The idea of the proof is that in order to be able to sort, you have to decide for every element to be sorted if it is larger or smaller to any other element to be sorted. By using transitivity this can be represented as a decision tree, which has n nodes and log n depth at best. So if you want to have performance better than Ω(n log n) this means removing edges from that decision tree. But if the decision tree is not complete, than how can you make sure that you have made a correct decision about some elements a and b?
Can you with no limitation on memory usage create such an algorithm?
So as from above that is not possible. And the remaining questions are therefore of no relevance.
If the integers are in a limited range then an O(n) "sort" of them would involve having a bit vector of "n" bits ... looping over the integers in question and setting the n%8 bit of offset n//8 in that byte array to true. That is an "O(n)" operation. Another loop over that bit array to list/enumerate/return/print all the set bits is, likewise, an O(n) operation. (Naturally O(2n) is reduced to O(n)).
This is a special case where n is small enough to fit within memory or in a file (with seek()) operations). It is not a general solution; but it is described in Bentley's "Programming Pearls" --- and was allegedly a practical solution to a real-world problem (involving something like a "freelist" of telephone numbers ... something like: find the first available phone number that could be issued to a new subscriber).
(Note: log(10*10) is ~24 bits to represent every possible integer up to 10 digits in length ... so there's plenty of room in 2*31 bits of a typical Unix/Linux maximum sized memory mapping).
I believe you are looking for radix sort.
Bucket sort and radix sort are close cousins; bucket sort goes from MSD to LSD, while radix sort can go in both "directions" (LSD or MSD). How do both algorithms work, and in particular how do they differ?
The initial pass of both RadixSort and BucketSort is exactly the same. The elements are put in buckets (or bins) of incremental ranges (e.g. 0-10, 11-20, ... 90-100), depending on the number of digits in the largest number.
In the next pass, however, BucketSort orders up these 'buckets' and appends them into one array. However, RadixSort appends the buckets without further sorting and 're-buckets' it based on the second digit (ten's place) of the numbers. Hence, BucketSort is more efficient for 'Dense' arrays, while RadixSort can handle sparse (well, not exactly sparse, but spaced-out) arrays well.
Bucket Sort and Radix Sort are like sister sorting algorithms because they are not comparison sorts and the general idea is similar. Also, they both are a bit abstract in implementation.
Radix Sort:
Radix means base(binary, octal, decimal,etc). Therefore, this sorting is for numbers (also used for strings). This works when each element E is like ek...e2e1e0, where ei is in some range. (usually 0 to a base like 0-9 in decimal or 0-255 in ASCII characters)
It then uses k passes of a stable sub-sorting algorithm (It has to be stable or else Radix sort won't work) to sort the numbers. This sub-sorting algorithm is usually Counting sort or Bucket sort as well but it cannot be Radix sort itself.
You can start from Most Significant Digit or Least Significant Digit because it shuffles every number in each pass (from k to 0 or 0 to k)
It is a stable sorting algorithm.
Bucket Sort:
It separates array into smaller groups or buckets and sorts them individually using a sub-sorting algorithm or recursive call to itself and then combines the result. For example, sorting players by adding into buckets of their team then sorting them by their jersey numbers or something like sorting numbers from ranging from 1-30 into 3 buckets of 1-10, 11-20, 21-30.
The combine step is trivial (unlike merge sort). just copy elements of each bucket back into original array or join the head of each bucket with tail of previous bucket (in case of linked lists)
Radix/base could be a type/instance of the group/bucket while sorting numbers. Therefore you could think of MSD radix as a modified instance of bucket sort
Bucket sort is not in-place but stable sorting algorithm. However, some variations of bucket sort might not be stable (if you use a sub-sorting algorithm which is not stable)