Based on this radix sort article http://www.geeksforgeeks.org/radix-sort/ I'm struggling to understand what is being explained in terms of the time complexity of certain methods in the sort.
From the link:
Let there be d digits in input integers. Radix Sort takes O(d*(n+b)) time where b is the base for representing numbers, for example, for decimal system, b is 10. What is the value of d? If k is the maximum possible value, then d would be O(log_b(k)). So overall time complexity is O((n+b)*logb(k)). Which looks more than the time complexity of comparison based sorting algorithms for a large k. Let us first limit k. Let k≤nc where c is a constant. In that case, the complexity becomes O(nlogb(n)).
So I do understand that the sort takes O(d*n) since there are d digits therefore d passes, and you have to process all n elements, but I lost it from there. A simple explanation would be really helpful.
Assuming we use bucket sort for the sorting on each digit: for each digit (d), we process all numbers (n), placing them in buckets for all possible values a digit may have (b).
We then need to process all the buckets, recreating the original list. Placing all items in the buckets takes O(n) time, recreating the list from all the buckets takes O(n + b) time (we have to iterate over all buckets and all elements inside them), and we do this for all digits, giving a running time of O(d * (n + b)).
This is only linear if d is a constant and b is not asymptotically larger than n. So indeed, if you have numbers of log n bits, it will take O(n log n) time.
Related
So assume we are given an array of m numbers, the max number in this array is k. There are duplicates in this array.
let array a = [1,2,3,1,2,5,1,2,3,4]
Is there an algorithm that prints out this array after o(n) operation result in [1,2,3,4,5](both sorted and no duplicate), where n is the quantity of unique values.
We are allowed to use k memory -- 5 in this case.
The algorithm I have in mind is to use a hash table. Insert value into a hash table, if the value exist before, we ignore it. This will sort automatically. However, if we have 5 number, [1,2,3,100,4] but one of them is 100, means when printing these 5 numbers, we need to run o(k) ~= 100 time instead of o(n) ~= 5 time.
Is there a way to solve this problem?
I don't think there exists such algorithm. Take a look here https://en.wikipedia.org/wiki/Sorting_algorithm
Essentially for comparison based algorithms the best you can achieve is O(nlogn). But since you have provided the max value k I would assume you want something more than just comparison based algorithm.
But for non-comparison based algorithms, since it by nature depends on magnitude of the numbers, the complexity has to reflect such dependency - meaning you will definitely have k somewhere in your total time complexity. You won't be able to find an algorithm of just O(n).
Conversly, if that O(n) algorithm were to exist and were not to depend on k. You can sort any array of n numbers since k is an extra, useless information.
You suggest that printing 5 numbers takes o(k) (or 100) time instead of o(n). That is wrong because, to print those 5 numbers, it takes 5 time to iterate and print. How would the value of your number change the time in which it takes to pull off this problem? The only situation when that should make a difference is if the value is greater than the allowable value in a 32-bit integer, or 2^32-1. Then you would have to detect those cases and treat them differently. However, assuming you don't have any integers of that size, you should be able to print 5 integers in O(5) time. I would go back over your calculation of the time it takes to go through your algorithm.
With your method, if you're using efficient algorithms, you should be able to remove duplicates in O(n log n) time, as seen here.
The way I see it, if you have a piece of the algorithm (the hashing part, where you remove duplicates and sort) running in O(n log n) time, and a piece of the algorithm (printing the array) running O(N) (or O(5) in this case), the entire algorithm runs in O(N) time: O(N) + O(N log N) -> O(N), since O(N) >= O(N log N). I hope that answers what you were asking for!
Looks like I was wrong, since O(N log N) of course grows faster than O(N). I don't think there's any way to pull off your problem.
So i came upon this question where:
we have to sort n numbers between 0 and n^3 and the answer of time complexity is O(n) and the author solved it this way:
first we convert the base of these numbers to n in O(n), therefore now we have numbers with maximum 3 digits ( because of n^3 )
now we use radix sort and therefore the time is O(n)
so i have three questions :
1. is this correct? and the best time possible?
2. how is it possible to convert the base of n numbers in O(n)? like O(1) for each number? because some previous topics in this website said its O(M(n) log(n))?!
3. and if this is true, then it means we can sort any n numbers from 0 to n^m in O(n) ?!
( I searched about converting the base of n numbers and some said its
O(logn) for each number and some said its O(n) for n numbers so I got confused about this too)
1) Yes, it's correct. It is the best complexity possible, because any sort would have to at least look at the numbers and that is O(n).
2) Yes, each number is converted to base-n in O(1). Simple ways to do this take O(m^2) in the number of digits, under the usual assumption that you can do arithmetic operations on numbers up to O(n) in O(1) time. m is constant so O(m^2) is O(1)... But really this step is just to say that the radix you use in the radix sort is in O(n). If you implemented this for real, you'd use the smallest power of 2 >= n so you wouldn't need these conversions.
3) Yes, if m is constant. The simplest way takes m passes in an LSB-first radix sort with a radix of around n. Each pass takes O(n) time, and the algorithm requires O(n) extra memory (measured in words that can hold n).
So the author is correct. In practice, though, this is usually approached from the other direction. If you're going to write a function that sorts machine integers, then at some large input size it's going to be faster if you switch to a radix sort. If W is the maximum integer size, then this tradeoff point will be when n >= 2^(W/m) for some constant m. This says the same thing as your constraint, but makes it clear that we're thinking about large-sized inputs only.
There is wrong assumption that radix sort is O(n), it is not.
As described on i.e. wiki:
if all n keys are distinct, then w has to be at least log n for a
random-access machine to be able to store them in memory, which gives
at best a time complexity O(n log n).
The answer is no, "author implementation" is (at best) n log n. Also converting these numbers can take probably more than O(n)
is this correct?
Yes it's correct. If n is used as the base, then it will take 3 radix sort passes, where 3 is a constant, and since time complexity ignores constant factors, it's O(n).
and the best time possible?
Not always. Depending on the maximum value of n, a larger base could be used so that the sort is done in 2 radix sort passes or 1 counting sort pass.
how is it possible to convert the base of n numbers in O(n)? like O(1) for each number?
O(1) just means a constant time complexity == fixed number of operations per number. It doesn't matter if the method chosen is not the fastest if only time complexity is being considered. For example, using a, b, c to represent most to least significant digits and x as the number, then using integer math: a = x/(n^2), b = (x-(a*n^2))/n, c = x%n (assumes x >= 0). (side note - if n is a constant, then an optimizing compiler may convert the divisions into a multiply and shift sequence).
and if this is true, then it means we can sort any n numbers from 0 to n^m in O(n) ?!
Only if m is considered a constant. Otherwise it's O(m n).
This is a problem from the Cormen text, but I'd like to see if there are any other solutions.
Given an array with n distinct numbers, you need to find the m largest ones in the array, and have
them in sorted order. Assume n and m are large, but grow differently. In particular, you need
to consider below the situations where m = t*n, where t is a small number, say 0.1, and then the
possibility m = √n.
The solution given in the book offers 3 options:
Sort the array and return the top m-long segment
Convert the array to a max-heap and extract the m elements
Select the m-th largest number, partition the array about it, and sort the segment of larger entries.
These all make sense, and they all have their pros and cons, but I'm wondering, is there another way to do it? It doesn't have to be better or faster, I'm just curious to see if this is a common problem with more solutions, or if we are limited to those 3 choices.
The time complexities of the three approaches you have mentioned are as follows.
O(n log n)
O(n + m log n)
O(n + m log m)
So option (3) is definitely better than the others in terms of asymptotic complexity, since m <= n. When m is small, the difference between (2) and (3) is so small it would have little practical impact.
As for other ways to solve the problem, there are infinitely many ways you could, so the question is somewhat poor in this regard. Another approach I can think of as being practically simple and performant is the following.
Extract the first m numbers from your list of n into an array, and sort it.
Repeatedly grab the next number from your list and insert it into the correct location in the array, shifting all the lesser numbers over by one and pushing one out.
I would only do this if m was very small though. Option (2) from your original list is also extremely easy to implement if you have a max-heap implementation and will work great.
A different approach.
Take the first m numbers, and turn them into a min heap. Run through the array, if its value exceeds the min of the top m then you extract the min value and insert the new one. When you reach the end of the array you can then extract the elements into an array and reverse it.
The worst case performance of this version is O(n log(m)) placing it between the first and second methods for efficiency.
The average case is more interesting. On average only O(m log(n/m)) of the elements are going to pass the first comparison test, each time incurring O(log(m)) work so you get O(n + m log(n/m) log(m)) work, which puts it between the second and third methods. But if n is many orders of magnitude greater than m then the O(n) piece dominates, and the O(n) median select in the third approach has worse constants than the one comparison per element in this approach, so in this case this is actually the fastest!
I heard that if we are sorting n numbers and that the numbers being sorted were converted to base n, then radix sort could be performed in O(n) time.
Have I got this right?
If so, how exactly is this achieved. If we are dealing with 5 numbers and we convert them all to base 5, we can separate the digits into 5 buckets (0, 1, 2, 3, 4).
Even if the numbers we were dealing with had only 7 digits max, wouldn't you still have to cycle through at least 7 * 5 times..? This doesn't seem right.. however.
Sorry, kind of confused about this.
Thanks for your help.
Radix sort works by picking some number base b, writing all the numbers in the input in base b, then sorting the numbers one digit at a time. In this answer, I'll focus on least-significant digit radix sort, in which we sort everything by the least-significant digit, then the second-least-significant digit, etc.
Each time we sort the numbers by some digit, we have to do O(n) work to distribute the elements across all the buckets, then O(n + b) work to iterate across the buckets and obtain the elements in sorted order. Therefore, the runtime for one round of radix sort is O(n + b).
The number of rounds of radix sort depends on the number of digits in each of the numbers. If you write numbers in base b, there will be O(logb M) base-b digits in the number M. If we let M denote the maximum number in the input array, the number of rounds of radix sort will then be O(logb M). Therefore, the asymptotic runtime of radix sort is
O(n + b) · O(logb M) = O((n + b) logb M).
In a typical binary radix sort, you'd pick b = 2 and get a runtime of O(n log M). However, you can choose b to be any value you'd like. If you pick a larger value of b, then there will be fewer base-b digits in each of the numbers (as an example, write a number in base-10 and then in base-16; you'll usually need fewer digits in base-16). In your original question, you asked
Even if the numbers we were dealing with had only 7 digits max, wouldn't you still have to cycle through at least 7 * 5 times?
The answer is "not necessarily." If you do a base-10 radix sort with 7-digit numbers, then yes, you'd have to cycle 7 times. However, if you used a base-100 radix sort, you'd only need to cycle 4 times.
Your other question was about using base-n for the radix sort. If we choose the base we use to be the number n, then we get that the runtime is
O((n + n) logn M) = O(n logn M) = O(n log M / log n)
(This uses the change-of-basis formula for logarithms to rewrite logn M = log M / log n).
This is not O(n), nor should you expect it to be. Think of it this way - radix sort's runtime depends on the length of the strings being sorted. If you sort a small number of extremely long numbers, the runtime is bound to be greater than the time to sort a small number of small numbers simply because you have to actually read the digits of the large number. The trick of using base n is simply a technique for speeding up the algorithm asymptotically.
Hope this helps!
Question:
Given n integers in the range [1, k], preprocesses its input and then
answers any query about how many of the n integers have values between a and b, where 1 ≤ a, b ≤ k
are two given parameters. Your algorithm should use O(n + k) preprocessing time.
Your algorithm is reasonably good, but it can be made much faster. Specifically, your algorithm has O(1) preprocessing time, but then spends O(n) time per query because of the linear cost of the time required to do the partitioning step.
Let's consider an alternative approach. Suppose that all of your values were in sorted order. In this case, you could find the number of elements in a range very quickly by just doing two binary searches - a first binary search to find the index of the lower bound, and a second search to find the upper bound - and could just subtract the indices. This would take time O(log n). If you can preprocess the input array to sort it in time O(n + k), then this approach will result in exponentially faster lookup times.
To do this sorting, as #minitech has pointed out, you can use the counting sort algorithm, which sorts in time O(n + k) for integers between 1 and k. Consequently, using both counting sort and the binary search together gives O(n + k) setup time and O(log n) query time.
If you are willing to trade memory for efficiency, though, you can speed this up even further. Let's suppose that k is a reasonably small number (say, not more than 100). Then if you are okay using O(k) space, you can answer these queries in O(1) time. The idea is as follows: build up a table of k elements that represents, for each element k, how many elements of the original array are smaller than k. If you have this array, you can find the total number of elements in some subrange by looking up how many elements are less than b and how many elements are less than a (each in O(1) time), then subtracting them.
Of course, to do this, you have to actually build up this table in time O(n + k). This can be done as follows. First, create an array of k elements, then iterate across the original n-element array and for each element increment the spot in the table corresponding to this number. When you're done (in time O(n + k)), you will have filled in this table with the number of times that each of the values in the range 1 - k exists in the original array (this is, incidentally, how counting sort works). Next, create a second table of k elements that will hold the cumulative frequency. Then, iterate across the histogram you built in the first step, and fill in the cumulative frequency table with the cumulative total number of elements encountered so far as you walk across the histogram. This last step takes time O(k), for a grand total of time O(n + k) for setup. You can now answer queries in time O(1).
Hope this helps!
Here is another simple algorithm:
First allocate an array A of size k, then iterate over n elements and for each integer x increment A[x] by one. this will take O(n) time.
Then compute prefix sum of array A, and store them as array B. this will take O(k).
now for any query of points(a, b) you can simply return: B[b]-B[a]+A[a]