I am having trouble understanding the difference between log(k) and log(n) in complexity analysis.
I have an array of size n. I have another number k < n that is an input of the algorithm (so it's not a constant known ahead of time). What are some examples of algorithms that would have log(n) vs those that would have log(k) in their complexity? I can only think of algorithms that have log(n) in their complexity.
For example, mergesort has log(n) complexity in its runtime analysis (O(nlogn)).
If your algorithm takes a list of size n and a number of magnitude k < n, the input size is on the order of n + log(k) (assuming k may be on the same asymptotic order of n). Why? k is a number represented in a place-value system (e.g., binary or decimal) and a number of magnitude k requires on the order of log k digits to represent.
Therefore, if your algorithm takes an input k and uses it in a way that requires all its digits be used or checked (e.g., equality is being checked, etc.) then the complexity of the whole algorithm is at least on the order of log k. If you do more complicated things with the number, the complexity could be even higher. For instance, if you have something like for i = 1 to k do ..., the complexity of your algorithm is at least k - maybe higher, since you're comparing to a log k-bit number k times (although i will use fewer bits than k for many/most values of i, depending on the base).
There's no "one-size-fits-all" explanation as to where an O(log k) term might come up.
You sometimes see this runtime arise in searching and sorting algorithms where you only need to rearrange some small part of the sequence. For example, the C++ standard library's std::partial_sort function, which rearranges the sequence so that the first k elements are in sorted order and the remainder are in arbitrary order in time O(n log k). One way this could be implemented is to maintain a min-heap of size at most k and do n insertions/deletions on it, which is n operations that each take time O(log k). Similarly, there's an O(n log k)-time algorithm for finding the k largest elements in a data stream, which works by maintaining a max-heap of at most k elements.
(Neither of these approaches are optimal, though - you can do a partial sort in time O(n + k log k) using a linear-time selection algorithm and can similarly find the top k elements of a data stream in O(n).)m
You also sometimes see this runtime in algorithms that involve a divide-and-conquer strategy where the size of the problem in question depends on some parameter of the input size. For example, the Kirkpatrick-Seidel algorithm for computing a convex hull does linear work per level in a recurrence, and the number of layers is O(log k), where k is the number of elements in the resulting convex hull. The total work is then O(n log k), making it an output-sensitive algorithm.
In some cases, an O(log k) term can arise because you are processing a collection of elements one digit at a time. For example, radix sort has a runtime of O(n log k) when used to sort n values that range from 0 to k, inclusive, with the log k term arising from the fact that there are O(log k) digits in the number k.
In graph algorithms, where the number of edges (m) is related to but can be independent of the number of nodes (n), you often see runtimes like O(m log n), as is the case if you implement Dijkstra's algorithm with a binary heap.
Related
Based on my research, I am gaining conflicting information about this simple algorithm. This algorithm is a basic matrix transposition, that transposes an n x n matrix A.
My current understanding is this algorithm would run at O(n^2) time and have a space complexity of O(1) as the matrix we manipulate would be the same one we deal with.
But- I have also been told it would actually run O(n) time and have space complexity of O(n) as well. Which means it wouldn't be in-place, as it requires extra space for manipulation.
Which thought process is correct here for the transpose algo below?
Transpose(A)
1. for i = 1 to n -1
2. for j = i + 1 to n
3. exchange A[i, j] with A[j,i]
Some confusion might arise from the facts that the workload is proportional to the number of elements in the array, and these elements occupy their own space. So by some notation abuse or inattention, both would be said "O(n)".
But this is wrong because
n is clearly not the number of elements but the number of rows/columns of the array;
by definition the space complexity does not include the input and output data, but any auxiliary space that would be required.
Hence we can confirm the time complexity O(n²) - in fact Θ(n²) - and space complexity O(1). The algorithm is in-place.
Final remark:
If we denote the number of elements as m, the time complexity is O(m), and there is no contradiction.
I have been learning about Radix sort recently and one of the sources I have used is the Wikipedia page. At the moment there is the following paragraph there regarding the efficiency of the algorithm:
The topic of the efficiency of radix sort compared to other sorting
algorithms is somewhat tricky and subject to quite a lot of
misunderstandings. Whether radix sort is equally efficient, less
efficient or more efficient than the best comparison-based algorithms
depends on the details of the assumptions made. Radix sort complexity
is O(wn) for n keys which are integers of word size w. Sometimes w is
presented as a constant, which would make radix sort better (for
sufficiently large n) than the best comparison-based sorting
algorithms, which all perform O(n log n) comparisons to sort n keys.
However, in general w cannot be considered a constant: if all n
keys are distinct, then w has to be at least log n for a random-access
machine to be able to store them in memory, which gives at best a time
complexity O(n log n). That would seem to make radix sort at most
equally efficient as the best comparison-based sorts (and worse if
keys are much longer than log n).
The part in bold has regrettably become a bit of a block that I am unable to get past. I understand that in general Radix sort is O(wn), and through other sources have seen how O(n) can be achieved, but cannot quite understand why n distinct keys requires O(n log n) time for storage in a random-access machine. I'm fairly certain it comes down to some simple mathematics, but unfortunately a solid understanding remains just beyond my grasp.
My closest attempt is as follows:
Given a base, 'B' and a number in that base, 'N', The maximum digits 'N' can have is:
(logB of N) + 1.
If each number in a given list, L, is unique, then we have up to:
L *((logB of N) + 1) possibilities
At which point I'm unsure how to progress.
Is anyone able to please expand on the above section in bold and break down why n distinct keys requires a minimum of log n for random-access storage?
Assuming MSB radix sort with constant m bins:
For an arbitrarily large data type which must accommodate at least n distinct values, the number of bits required is N = ceiling(log2(n))
Thus the amount of memory required to store each value is also O(log n); assuming sequential memory access, the time complexity of reading / writing a value is O(N) = O(log n), although can use pointers instead
The number of digits is O(N / m) = O(log n)
Importantly, each consecutive digit must differ by a power-of-2, i.e. m must also be a power-of-2; assume this to be small enough for the HW platform, e.g. 4-bit digits = 16 bins
During sorting:
For each radix pass, of which there are O(log n):
Count each bucket: get the value of the current digit using bit operations - O(1) for all n values. Should note that each counter must also be N bits, although increments by 1 will be (amortized) O(1). If we had used non-power-of-2 digits, this would in general be O(log n log log n) ( source )
Make the bucket count array cumulative: must perform m - 1 additions, each of which is O(N) = O(log n) (unlike the increment special case)
Write the output array: loop through n values, determine the bin again, and write the pointer with the correct offset
Thus the total complexity is O(log n) * [ n * O(1) + m * O(log n) + n * O(1) ] = O(n log n).
Was reading CLRS when I encountered this:
Why do we not ignore the constant k in the big o equations in a. , b. and c.?
In this case, you aren't considering the run time of a single algorithm, but of a family of algorithms parameterized by k. Considering k lets you compare the difference between sorting n/n == 1 list and n/2 2-element lists. Somewhere in the middle, there is a value of k that you want to compute for part (c) so that Θ(nk + n lg(n/k)) and Θ(n lg n) are equal.
Going into more detail, insertion sort is O(n^2) because (roughly speaking) in the worst case, any single insertion could take O(n) time. However, if the sublists have a fixed length k, then you know the insertion step is O(1), independent of how many lists you are sorting. (That is, the bottleneck is no longer in the insertion step, but the merge phase.)
K is not a constant when you compare different algorithms with different values of k.
Are there any famous algorithms with this complexity?
I was thinking maybe a skip list where levels of the nodes are not determined by the number of tails coin tosses, but instead are use a number generated randomly (with uniform distribution) from the (1,log(n)) period to determine the level of the node. Such a data structure would have a find(x) operation with the complexity of O(n/log(n)) (I think, at least). I was curious whether there was anything else.
It's common to see algorithms whose runtime is of the form O(nk / log n) or O(log n / log log n) when using the method of Four Russians to speed up an existing algorithm. The classic Four Russians speedup reduces the cost of doing a matrix/vector product on Boolean matrices from O(n2) to O(n2 / log n). The standard dynamic programming algorithm for sequence alignment on two length-n strings runs in time O(n2), which can be decreased to O(n2 / log n) by using a similar trick.
Similarly, the prefix parity problem - in which you need to maintain a sequence of Boolean values while supporting the "flip" and "parity of the prefix of a sequence" operations can be solved in time O(log n / log log n) by using a Four-Russians speedup. (Notice that if you express the runtime as a function of k = log n, this is O(k / log k).
The background
According to Wikipedia and other sources I've found, building a binary heap of n elements by starting with an empty binary heap and inserting the n elements into it is O(n log n), since binary heap insertion is O(log n) and you're doing it n times. Let's call this the insertion algorithm.
It also presents an alternate approach in which you sink/trickle down/percolate down/cascade down/heapify down/bubble down the first/top half of the elements, starting with the middle element and ending with the first element, and that this is O(n), a much better complexity. The proof of this complexity rests on the insight that the sink complexity for each element depends on its height in the binary heap: if it's near the bottom, it will be small, maybe zero; if it's near the top, it can be large, maybe log n. The point is that the complexity isn't log n for every element sunk in this process, so the overall complexity is much less than O(n log n), and is in fact O(n). Let's call this the sink algorithm.
The question
Why isn't the complexity for the insertion algorithm the same as that of the sink algorithm, for the same reasons?
Consider the actual work done for the first few elements in the insertion algorithm. The cost of the first insertion isn't log n, it's zero, because the binary heap is empty! The cost of the second insertion is at worst one swap, and the cost of the fourth is at worst two swaps, and so on. The actual complexity of inserting an element depends on the current depth of the binary heap, so the complexity for most insertions is less than O(log n). The insertion cost doesn't even technically reach O(log n) until after all n elements have been inserted [it's O(log (n - 1)) for the last element]!
These savings sound just like the savings gotten by the sink algorithm, so why aren't they counted the same for both algorithms?
Actually, when n=2^x - 1 (the lowest level is full), n/2 elements may require log(n) swaps in the insertion algorithm (to become leaf nodes). So you'll need (n/2)(log(n)) swaps for the leaves only, which already makes it O(nlogn).
In the other algorithm, only one element needs log(n) swaps, 2 need log(n)-1 swaps, 4 need log(n)-2 swaps, etc. Wikipedia shows a proof that it results in a series convergent to a constant in place of a logarithm.
The intuition is that the sink algorithm moves only a few things (those in the small layers at the top of the heap/tree) distance log(n), while the insertion algorithm moves many things (those in the big layers at the bottom of the heap) distance log(n).
The intuition for why the sink algorithm can get away with this that the insertion algorithm is also meeting an additional (nice) requirement: if we stop the insertion at any point, the partially formed heap has to be (and is) a valid heap. For the sink algorithm, all we get is a weird malformed bottom portion of a heap. Sort of like a pine tree with the top cut off.
Also, summations and blah blah. It's best to think asymptotically about what happens when inserting, say, the last half of the elements of an arbitrarily large set of size n.
While it's true that log(n-1) is less than log(n), it's not smaller by enough to make a difference.
Mathematically: The worst-case cost of inserting the i'th element is ceil(log i). Therefore the worst-case cost of inserting elements 1 through n is sum(i = 1..n, ceil(log i)) > sum(i = 1..n, log i) = log 1 + log 1 + ... + log n = log(1 × 2 × ... × n) = log n! = O(n log n).
Ran into the same problem yesterday. I tried coming up with some form of proof to satisfy myself. Does this make any sense?
If you start inserting from the bottom, The leaves will have constant time insertion- just copying it into the array.
The worst case running time for a level above the leaves is:
k*(n/2h)*h
where h is the height (leaves being 0, top being log(n) ) k is a constant( just for good measure ). So (n/2h) is the number of nodes per level and h is the MAXIMUM number of 'sinking' operations per insert
There are log(n) levels,
Hence, The total running time will be
Sum for h from 1 to log(n): n* k* (h/2h)
Which is k*n * SUM h=[1,log(n)]: (h/2h)
The sum is a simple Arithmetico-Geometric Progression which comes out to 2.
So you get a running time of k*n*2, Which is O(n)
The running time per level isn't strictly what i said it was but it is strictly less than that.Any pitfalls?