This question already has answers here:
Space requirements of a merge-sort
(2 answers)
Closed 2 years ago.
At first look, it makes sense that merge sort has space complexity of O(n) because to sort the unsorted array I'm splitting and creating subarrays but the sum of sizes of all the subarray will be n.
Question : The main concern that I have is that of memeory allocation of mergerSort() function during recurssion. I have a main stack, and each function call to mergerSort() ( recussively) will be pushed on the stack. Now each recussively called mergeSort() function will have its own stack. Therefore, say if we have made 5 recussive calls to mergeSort() then the main stack will contain 5 function call where each function call will have its own function stack. Now each function stack will have its own local varibales like left subarray and right subarray that the function creates. Hence, each of the 5 function stacks should have 5 different subarrays in memeory. So shouldn't the space grow with the growth in recussive calls ?
Memory should be linear
Although each call to mergeSort triggers two recursive calls, so it makes sense to talk about and draw the binary tree of recursive calls, only one of those two recursive calls is performed at a time; the first call ends before the second call starts. Hence, at any given time, only one branch of the tree is being explored. The "call stack" represents this branch.
The depth of the recursion tree is at most log(n), therefore the height of the call stack is at most log(n).
How much memory does it take to explore one branch? In other words, how much memory is allocated on the call stack, at most, at any given time?
At the bottom of the call stack, there is an array of size n.
On top of that is an array of size n/2.
On top of that is an array of size n/4.
Etc...
So the total size of the call stack is at most n + n/2 + n/4 + ... < 2n.
Hence the total size of the call stack is at most 2n.
Possible memory leak
If your implementation of merge sort allocates a new array at every recursive call, and you forget to free those arrays at the end of the call, then the total allocated memory becomes the total memory required for the whole tree, instead of just one branch.
Consider all the nodes at a given depth in the tree. The subarrays of these nodes add up to form the whole array. For instance, the root of the tree has an array of length n; then one level below that, there are two subarrays representing two halves of the original array; then one level below that, there are four subarrays representing four fourth of the original array; etc. Hence each level of the tree requires memory n. There are log(n) levels to the tree. Hence the total amount of memory allocated for the whole tree would be n log(n).
Conclusion
If merge sort has no memory leaks, then its space complexity is linear O(n). In addition, it is possible (although not always desirable) to implement merge sort in-place, in which case the space complexity is constant O(1) (all operations are performed directly inside the input array).
However, if your implementation of merge sort has a memory leak, i.e., you keep allocating new arrays in recursive calls, but do not free them when the recursive call returns, then it could easily have space complexity O(n log n).
Related
def quick_sort(array):
if len(array) <=1:
return array
pivot = array[-1]
array.pop()
less = []
greater = []
for num in array:
if num > pivot:
greater.append(num)
else:
less.append(num)
return quick_sort(less) + [pivot] + quick_sort(greater)
What's the space complexity of this implementation of quicksort? I just picked the last element as the pivot, created an array of the elements greater and the elements lesser and moved them accordingly. Then I recursively did that for both the lesser and greater arrays. So at the end, I'd have [pivot] + [pivot] + [pivot]... all in sorted order. Now I'm kind of confused about the space complexity. I have two sub arrays for the lesser and greater and also there's the recursion call stack. What do you think?
The space complexity of your implementation of quicksort is Θ(n2) in the worst case and Θ(n) on average.
Here’s how to see this. Imagine drawing out the full recursion tree for your algorithm. At any one point in time, the algorithm is in one of those recursive calls, with space needed to store all the data from that recursive call, plus all the space for the recursive calls above it. That’s because the call stack, at any one point in time, is a path from some call back up to the root call. Therefore, the space complexity is the maximum amount of space used on any path from a leaf in the recursion tree back up to the root.
Imagine, then, that you happen to pick the absolute worst pivot possible at each step - say, you always pick the smallest or largest element. Then your recursion tree is essentially a giant linked list, where the root holds an array of length n, under that is an array of length n-1, under that is an array of length n-2, etc. until you’re down to an array of length one. The space usage then is 1+2+3+...+n, which is Θ(n2). That’s not great.
On the other hand, suppose you’re looking at a more “typical” run of quicksort, in which you generally get good pivots. In that case, you’d expect that, about half the time, you get a pivot in the middle 50% of the array. With a little math, you can show that this means that, on expectation, you’ll have about two splits before the array size drops to 75% of its previous size. That makes the depth of the recursion tree O(log n). You’ll then have about two layers with arrays of size roughly n, about two layers with arrays of size roughly .75n, about two layers of size roughly (.75)2n, etc. That makes your space usage roughly
2(n + .75n + (.75)2n + ...)
= 2n(1 + .75 + (.75)2 + ...)
= Θ(n).
That last step follows because that’s the sum of a geometric series, which converges to some constant.
To improve your space usage, you’ll need to avoid creating new arrays at each level for your lesser and greater elements. Consider using an in-place partition algorithm to modify the array in place. If you’re clever, you can use that approach and end up with O(log n) total space usage.
Hope this helps!
I am having a hard time understanding what is O(1) space complexity. I understand that it means that the space required by the algorithm does not grow with the input or the size of the data on which we are using the algorithm. But what does it exactly mean?
If we use an algorithm on a linked list say 1->2->3->4, to traverse the list to reach "3" we declare a temporary pointer. And traverse the list until we reach 3. Does this mean we still have O(1) extra space? Or does it mean something completely different. I am sorry if this does not make sense at all. I am a bit confused.
To answer your question, if you have a traversal algorithm for traversing the list which allocate a single pointer to do so, the traversal algorithms is considered to be of O(1) space complexity. Additionally, let's say that traversal algorithm needs not 1 but 1000 pointers, the space complexity is still considered to be O(1).
However, if let's say for some reason the algorithm needs to allocate 'N' pointers when traversing a list of size N, i.e., it needs to allocate 3 pointers for traversing a list of 3 elements, 10 pointers for a list of 10 elements, 1000 pointers for a list of 1000 elements and so on, then the algorithm is considered to have a space complexity of O(N). This is true even when 'N' is very small, eg., N=1.
To summarise the two examples above, O(1) denotes constant space use: the algorithm allocates the same number of pointers irrespective to the list size. In contrast, O(N) denotes linear space use: the algorithm space use grows together with respect to the input size.
It is just the amount of memory used by a program. the amount of computer memory that is the main memory required by the algorithm to complete its execution with respect to the input size.
Space Complexity(s(P)) of an algorithm is the total space taken by the algorithm to complete its execution with respect to the input size. It includes both Constant space and Auxiliary space.
S(P) = Constant space + Auxiliary space
Constant space is the one that is fixed for that algorithm, generally equals to space used by input and local variables. Auxiliary Space is the extra/temporary space used by an algorithm.
Let's say I create some data structure with a fixed size, and no matter what I do to the data structure, it will always have the same fixed size. Operations performed on this data structure are therefore O(1).
An example, let's say I have an array of fixed size 100. Any operation I do, whether that is reading from the array or updating an element, that operation will be O(1) on the array. The array's size (and thus the amount of memory it's using) is not changing.
Another example, let's say I have a LinkedList to which I add elements to it. Every time I add an element to the LinkedList, that is a O(N) operation to the list because I am growing the amount of memory required to hold all of it's elements together.
Hope this helps!
O(n) complexity means merge sort in worst case takes a memory space equal to the number of elements present in the initial array. But hasn't it created new arrays while making the recursive calls? How that space is not counted?
A worst case implementation of top down merge sort could take more space than the original array, if it allocates both halves of the array in mergesort() before making the recursive calls to itself.
A more efficient top down merge sort uses an entry function that does a one time allocation of a temp buffer, passing the temp buffer's address as a parameter to one of a pair of mutually recursive functions that generate indices and merge data between the two arrays.
In the case of a bottom up merge sort, a temp array 1/2 the size of the original array could be used, merging both halves of the array, ending up with the first half of data in the temp array, and the second half in the original array, then doing a final merge back into the original array.
However the space complexity is O(n) in either case, since constants like 2 or 1/2 are ignored for big O.
MergeSort has enough with a single buffer of the same size as the original array.
In the usual version, you perform a merge from the array to the extra buffer and copy back to the array.
In an advanced version, you perform the merges from the array to the extra buffer and conversely, alternately.
Note: This answer is wrong, as was pointed out to me in the comments. I leave it here as I believe it is helpful to most people who wants to understand these things, but remember that this algorithm is actually called in-place mergesort and can have a different runtime complexity than pure mergesort.
Merge sort is easy to implement to use the same array for everything, without creating new arrays. Just send the bounds in each recursive call. So something like this (in pseudocode):
mergesort(array) ->
mergesort'(array, 0, length of array - 1)
mergesort'(array, start, end) ->
mergesort'(array, start, end/2)
mergesort'(array, end/2+1, end)
merge(array, start, end/2, end/2+1, end)
merge(array, start1, end1, start2, end2) ->
// This function merges the two partitions
// by just moving elements inside array
In Merge Sort, space complexity is always omega(n) as you have to store the elements somewhere. Additional space complexity can be O(n) in an implementation using arrays and O(1) in linked list implementations. In practice implementations using lists need additional space for list pointers, so unless you already have the list in memory it shouldn't matter. edit if you count stack frames, then it's O(n)+ O(log n) , so still O(n) in case of arrays. In case of lists it's O(log n) additional memory.
That's why in merge-sort complexity analysis people mention 'additional space requirement' or things like that. It's obvious that you have to store the elements somewhere, but it's always better to mention 'additional memory' to keep purists at bay.
Proposition . In the resizing array implementation of Stack,
the average number of array accesses for any sequence of operations starting from
an empty data structure is constant in the worst case.
Proof sketch: For each push() that causes the array to grow ( say from size N to
size 2N), consider the N/2 - 1 push() operations that most recently caused the
stack size to grow to k, for k from N/2 + 2 to N. Averaging the 4N array accesses to
grow the array with N/2 array accesses (one for each push), we get an average cost
of 9 array accesses per operation. Proving that the number of array accesses used by
any sequence of M operations is proportional to M is more intricate.
(Algorithms 4th Edition Chapter 1.4)
I didn't understand the Proof Sketch Completely. Please help me in getting this understand.
I think this is sort of amortized analysis where you charge requests like push() for work that isn't directly due to them, and then show that nobody has to pay too high a bill, which means that the average cost of work done is small.
In this case you have to copy the entire array when you run out of space, but you double the size when you do this, so you don't copy very often - e.g. at size 1, 2, 4, 8, 16... Here we bill each array copy to the push() operations which have been done since the last array copy. This means that if you do nothing but push() then each push() gets the bill only for the first array copy that occurs after it, so if the bill (split over a number of push() operations) is small per push() then the amortized cost is small.
If the array is of size N before it runs out of space and gets doubled in size then this article says this costs 4N operations, which sounds reasonable, and we don't care about constant factors much anyway. This gets split over all the operations since the last doubling. The last doubling was from size N/2 to size N so there are about N/2 of them. This gets you 4N ops split over N/2 push() operations so each push gets a shared bill of 8. Don't forget that a push() involves an array write whether or not it triggers a size-doubling and you get an average cost of 9 writes per push().
Is there an efficient algorithm for merging 2 max-heaps that are stored as arrays?
It depends on what the type of the heap is.
If it's a standard heap where every node has up to two children and which gets filled up that the leaves are on a maximum of two different rows, you cannot get better than O(n) for merge.
Just put the two arrays together and create a new heap out of them which takes O(n).
For better merging performance, you could use another heap variant like a Fibonacci-Heap which can merge in O(1) amortized.
Update:
Note that it is worse to insert all elements of the first heap one by one to the second heap or vice versa since an insertion takes O(log(n)).
As your comment states, you don't seem to know how the heap is optimally built in the beginning (again for a standard binary heap)
Create an array and put in the elements of both heaps in some arbitrary order
now start at the lowest level. The lowest level contains trivial max-heaps of size 1 so this level is done
move a level up. When the heap condition of one of the "sub-heap"s gets violated, swap the root of the "sub-heap" with it's bigger child. Afterwards, level 2 is done
move to level 3. When the heap condition gets violated, process as before. Swap it down with it's bigger child and process recursively until everything matches up to level 3
...
when you reach the top, you created a new heap in O(n).
I omit a proof here but you can explain this since you have done most of the heap on the bottom levels where you didn't have to swap much content to re-establish the heap condition. You have operated on much smaller "sub heaps" which is much better than what you would do if you would insert every element into one of the heaps => then, you willoperate every time on the whole heap which takes O(n) every time.
Update 2: A binomial heap allows merging in O(log(n)) and would conform to your O(log(n)^2) requirement.
Two binary heaps of sizes n and k can be merged in O(log n * log k) comparisons. See
Jörg-R. Sack and Thomas Strothotte, An algorithm for merging heaps, Acta Informatica 22 (1985), 172-186.
I think what you're looking for in this case is a Binomial Heap.
A binomial heap is a collection of binomial trees, a member of the merge-able heap family. The worst-case running time for a union (merge) on 2+ binomial heaps with n total items in the heaps is O(lg n).
See http://en.wikipedia.org/wiki/Binomial_heap for more information.