Kindly explain with a simple problem
I have been going through a book where it is said that ArrayList will double after it reaches its limit and if the ArrayList is full it takes O(N) insertion time for inserting N element.Kindly explain by taking an ArrayList of few elements
Amortised time explained in simple terms:
If you do an operation say a million times, you don't really care about the worst-case or the best-case of that operation - what you care about is how much time is taken in total when you repeat the operation a million times.
So it doesn't matter if the operation is very slow once in a while, as long as "once in a while" is rare enough for the slowness to be diluted away. Essentially amortised time means "average time taken per operation, if you do many operations". Amortised time doesn't have to be constant; you can have linear and logarithmic amortised time or whatever else.
Let's take the example of a dynamic array, to which you repeatedly add new items. Normally adding an item takes constant time (that is, O(1)). But each time the array is full, you allocate twice as much space, copy your data into the new region, and free the old space. Assuming allocates and frees run in constant time, this enlargement process takes O(n) time where n is the current size of the array.
So each time you enlarge, you take about twice as much time as the last enlarge. But you've also waited twice as long before doing it! The cost of each enlargement can thus be "spread out" among the insertions. This means that in the long term, the total time taken for adding m items to the array is O(m), and so the amortised time (i.e. time per insertion) is O(1).
In the Amortized analysis.
Average case running time: average over all possible inputs for one algorithm (operation).
If using probability, called expected running time
What is the different between expected time and Amortized time?
Expected time:
We make some assumptions and, based on these assumptions, we make statements about the running time.
Hash tables is one such example. We assume that the data is well-distributed, and claim that the running time of operations are O(1), whereas the worst-case running time for an operation is actually O(n).
Amortized time:
Even though one operation may take longer than some given time, the time across multiple operations will balance out to give the mentioned running time.
(Well-implemented) self-resizing arrays is one such example. When you insert, it takes O(n) to resize the array, but, across many inserts, each will take O(1) on average.
Can someone explain amortized complexity in layman's terms? I've been having a hard time finding a precise definition online and I don't know how it entirely relates to the analysis of algorithms. Anything useful, even if externally referenced, would be highly appreciated.
Amortized complexity is the total expense per operation, evaluated over a sequence of operations.
The idea is to guarantee the total expense of the entire sequence, while permitting individual operations to be much more expensive than the amortized cost.
Example:
The behavior of C++ std::vector<>. When push_back() increases the vector size above its pre-allocated value, it doubles the allocated length.
So a single push_back() may take O(N) time to execute (as the contents of the array are copied to the new memory allocation).
However, because the size of the allocation was doubled, the next N-1 calls to push_back() will each take O(1) time to execute. So, the total of N operations will still take O(N) time; thereby giving push_back() an amortized cost of O(1) per operation.
Unless otherwise specified, amortized complexity is an asymptotic worst-case guarantee for any sequence of operations. This means:
Just as with non-amortized complexity, the big-O notation used for amortized complexity ignores both fixed initial overhead and constant performance factors. So, for the purpose of evaluating big-O amortized performance, you can generally assume that any sequence of amortized operations will be "long enough" to amortize away a fixed startup expense. Specifically, for the std::vector<> example, this is why you don't need to worry about whether you will actually encounter N additional operations: the asymptotic nature of the analysis already assumes that you will.
Besides arbitrary length, amortized analysis doesn't make assumptions about the sequence of operations whose cost you are measuring -- it is a worst-case guarantee on any possible sequence of operations. No matter how badly the operations are chosen (say, by a malicious adversary!), an amortized analysis must guarantee that a long enough sequence of operations may not cost consistently more than the sum of their amortized costs. This is why (unless specifically mentioned as a qualifier) "probability" and "average case" are not relevant to amortized analysis -- any more than they are to an ordinary worst-case big-O analysis!
In an amortized analysis, the time required to perform a sequence of data-structure operations is averaged over all the operations performed... Amortized analysis differs from average-case analysis in that probability is not involved; an amortized analysis guarantees the average performance of each operation in the worst case.
(from Cormen et al., "Introduction to Algorithms")
That might be a bit confusing since it says both that the time is averaged, and that it's not an average-case analysis. So let me try to explain this with a financial analogy (indeed, "amortized" is a word most commonly associated with banking and accounting.)
Suppose that you are operating a lottery. (Not buying a lottery ticket, which we'll get to in a moment, but operating the lottery itself.) You print 100,000 tickets, which you will sell for 1 currency unit each. One of those tickets will entitle the purchaser to 40,000 currency units.
Now, assuming you can sell all the tickets, you stand to earn 60,000 currency units: 100,000 currency units in sales, minus the 40,000 currency unit prize. For you, the value of each ticket is 0.60 currency units, amortized over all the tickets. This is a reliable value; you can bank on it. If you get tired of selling the tickets yourself, and someone comes along and offers to sell them for 0.30 currency units each, you know exactly where you stand.
For the lottery purchaser, the situation is different. The purchaser has an expected loss of 0.60 currency units when they purchase a lottery ticket. But that's probabilistic: the purchaser might buy ten lottery tickets every day for 30 years (a bit more than 100,000 tickets) without ever winning. Or they might spontaneously buy a single ticket one day, and win 39,999 currency units.
Applied to datastructure analysis, we're talking about the first case, where we amortize the cost of some datastructure operation (say, insert) over all the operations of that kind. Average-case analysis deals with the expected value of a stochastic operation (say, search), where we cannot compute the total cost of all the operations, but we can provide a probabilistic analysis of the expected cost of a single one.
It's often stated that amortized analysis applies to the situation where a high-cost operation is rare, and that's often the case. But not always. Consider, for example, the so-called "banker's queue", which is a first-in-first-out (FIFO) queue, made out of two stacks. (It's a classic functional data-structure; you can build cheap LIFO stacks out of immutable single-linked nodes, but cheap FIFOs are not so obvious). The operations are implemented as follows:
put(x): Push x on the right-hand stack.
y=get(): If the left-hand stack is empty:
Pop each element off the right-hand stack and
push it onto the left-hand stack. This effectively
reverses the right-hand stack onto the left-hand stack.
Pop and return the top element of the left-hand stack.
Now, I claim that the amortized cost of put and get is O(1), assuming that I start and end with an empty queue. The analysis is simple: I always put onto the right-hand stack, and get from the left-hand stack. So aside from the If clause, each put is a push, and each get is a pop, both of which are O(1). I don't know how many times I will execute the If clause -- it depends on the pattern of puts and gets -- but I know that every element moves exactly once from the right-hand stack to the left-hand stack. So the total cost over the entire sequence of n puts and n gets is: n pushes, n pops, and n moves, where a move is a pop followed by a push: in other words, the 2n operations (n puts and n gets) result in 2n pushes and 2n pops. So the amortized cost of a single put or get is one push and one pop.
Note that banker's queues are called that precisely because of the amortized complexity analysis (and the association of the word "amortized" with finance). Banker's queues are the answer to what used to be a common interview question, although I think it's now considered too well-known: Come up with a queue which implements the following three operations in amortized O(1) time:
1) Get and remove the oldest element of the queue,
2) Put a new element onto the queue,
3) Find the value of the current maximum element.
The principle of "amortized complexity" is that although something may be quite complex when you do it, since it's not done very often, it's considered "not complex". For example, if you create a binary tree that needs balancing from time to time - say once every 2^n insertions - because although balancing the tree is quite complex, it only happens once in every n insertions (e.g once at insertion number 256, then again at 512th, 1024th, etc). On all other insertions, the complexity is O(1) - yes, it takes O(n) once every n insertions, but it's only 1/n probability - so we multiply O(n) by 1/n and get O(1). So that is said to be "Amortized complexity of O(1)" - because as you add more elements, the time consumed for rebalancing the tree is minimal.
Amortized means divided over repeated runs. The worst-case behavior is guaranteed not to happen with much frequency. For example if the slowest case is O(N), but the chance of that happening is just O(1/N), and otherwise the process is O(1), then the algorithm would still have amortized constant O(1) time. Just consider the work of each O(N) run to be parceled out to N other runs.
The concept depends on having enough runs to divide the total time over. If the algorithm is only run once, or it has to meet a deadline each time it runs, then the worst-case complexity is more relevant.
Say you are trying to find the kth smallest element of an unsorted array.
Sorting the array would be O(n logn).
So then finding the kth smallest number is just locating the index, so O(1).
Since the array is already sorted, we never have to sort again. We will never hit the worst case scenario more than once.
If we perform n queries of trying to locate kth smallest, it will still be O(n logn) because it dominates over O(1). If we average the time of each operation it will be:
(n logn)/n or O(logn). So, Time Complexity/ Number of Operations.
This is amortized complexity.
I think this is how it goes, im just learning it too..
It is somewhat similar to multiplying worst case complexity of different branches in an algorithm with the probability of executing that branch, and adding the results. So if some branch is very unlikely to be taken, it contributes less to the complexity.
I think I am starting to understand at least the theory behind big Oh notation, i.e. it is a way of measuring the rate at which the speed of a function grows. In other words, big O quantifies an algorithm's efficiency. But the implementation of it is something else.
For example, in the best case scenario push and pull operations will be O(1) because the number of steps it takes to remove from or add to the stack are going to be fixed. Regardless of the value, the process will be the same.
I'm trying to envision how a sequence of events such as push and pop can degrade performance from O(1) to O(n^2). If I have an array of n/2 capacity, n push and pop operations, and a dynamic array that doubles or halves its capacity when full or half full, how is it possible that the sequence in which these operations occur can affect the speed in which a program completes? Since push and pop work on the top element of the stack, I'm having trouble seeing how efficiency goes from a constant to O(n^2).
Thanks in advance.
You're assuming that the dynamic array does its resize operations quite intelligently. If this is not the case, however, you might end up with O(n^2) runtime: Suppose the array does not double its size when full but simply is resized to size+1. Also, suppose it starts with size 1. You'd insert the first element in O(1). When inserting the second elment, the array would need to be resized to size 2, requiring it to copy the previous value. When inserting element k, it would currently have size k-1, and need to be resized to size k, resulting in k-1 elements that need to be copied, and so on.
Thus, for inserting n elements, you'd end up with copying the array n-1 times: O(n) resizes. The copy operations are also linearly dependent on n since the more elements are have been inserted, the more need to be copied: O(n) copies per resize. This results in O(n*n) = O(n^2) as its runtime complexity.
If I implement a stack as (say) a linked list, then pushes and pops will always be constant time (i.e. O(1)).
I would not choose a dynamic array implementation for a stack, unless runtime wasn't an issue for me, I happened to have a dynamic array ready-built and available to use, and I didn't have a more efficient stack implementation handy. However, if I did use an array that resized up or down when it became full or half-empty respectively, its runtime would be O(1) while the numbers of pushes and pops are low enough not to trigger the resize and O(n) when there is a resize (hence overall O(n)).
I can't think of a case where a dynamic array used as a stack could deliver performance as bad as O(n^2) unless there was a bug in its implementation.
This is regarding amortized analysis. Following is text from an article.
Amortized analyis for problems in which one must perform a series of
operations, and our goal is to analyze the time per operation. The
motivation for amortized analysis is that looking at the worst case
time per operation can be too pessimistic if the only way to produce
an expensive is to "set it up" with a a large number of cheap
operations before hand.
Question: What does author mean by last statement i.e., "if the only way to produce an expensive is to "set it up" with a a large number of cheap operations before
hand"? Can any one please explain with example what this statement mean?
Thanks!
Another example. Consider an array that dynamically increases it's capacity when an element is added that exceeds the current capacity. Let increasing the capacity be O(n), where n is the old size of the array. Now, adding an element has a worst case complexity of O(n), because we might have to increase the capacity. The idea behind amortized analysis is that you have to do n simple adds that cost O(1) before the capacity is exhausted. Thus, many cheap operations lead up to one expensive operation. In other words, the expensive operation is amortized by the cheap operations.
The author means that the only way an expensive operation can occour is to be preceded by a big number of cheap operations.
Look at this example:
We have a stack and we want the stack to implement in addition to the usual operations, also a operation called multipop(k) that pop k elements. Now, multipop costs O(min(n, k)), where n is the size of the stack; thus the prerequisite for the multipop to costs for example O(k) is to be precided by at least k cheap push each costing O(1).