Dijkstra and one question on analysis, is it related to implementation? - algorithm

What is the amortized update cost for each vertex in Dijkstra algorithm:
Answer: O(|E| / |V|)
My challenge is that:
this answer relate to implementation or not? i.e: fib heap or... and
next this relate to which operation of Dijkstra algorithm? I means
decrease key? remove min part? which part of analysis?
consider for example this pesudocode just for discuss (or any others):

Echoing #Paul, this question doesn't make a lot of sense because Dijkstra's algorithm isn't a data structure with many possible call sequences; it just does its thing and then exits.
If we assume a Fibonacci heap, whose operations has amortized time bounds, the amortized work done by the heap inside the loop for a vertex u is O(degree(u)), since Fibonacci heaps have amortized constant-time decrease key operations.
If you really want, you can further amortize the cost by defining an additional potential function on the order of (sum of degrees of nodes in the heap)/2 − (number of nodes in the heap times |E|/|V|). This potential function goes from zero to zero, and it makes the amortized cost of each decreasekey O(|E|/|V|).

Related

Why naive implementation of Dijkstra's shortest path algorithm takes O(nm) time

I'm watching the coursera lectures about algorithm and the professor introduces that the naive implementation of Dijkstra's shortest path algorithm, without using heaps, takes O(nm) time(n is the number of vertices, and m is the number of edges)
It claims that the main loop will go through the rest vertices besides the source, which are n-1 vertices, this I can understand, but inside the loop, the algorithm will go through edges that with tail in the processed vertices and head in the unprocessed vertices, to minimize the next path. But why does it mean there are m edges to go through, we can just go through the edges that qualifies the criteria(tail in the processed vertices and head in the unprocessed vertices) even in a naive implementation right?
Could anyone please help me understand this? thanks.
When you consider big-O time complexity, you should think of it as a form of upper-bound. Meaning, if some input can possibly make your program run in O(N) while some other input can possibly make your program run in O(NM), we generally say that this is an O(NM) program (especially when dealing with algorithms). This is called considering an algorithm in its worst-case.
There are rare cases where we don't consider the worst-case (amortized time complexity analysis, time complexity with random elements to it such as quicksort). In the case of quicksort, for example, the worst-case time complexity is O(N^2), but the chance of that happening is so, so small that in practice, if we select a random pivot, we can expect O(N log N) time complexity. That being said, unless you can guarantee (or have large confidence that) your program runs faster than O(NM), it is an O(NM) program.
Let's consider the case of your naive implementation of Dijkstra's algorithm. You didn't specify whether you were considering directed or undirected graphs, so I'll assume that the graph is undirected (the case for a directed graph is extremely similar).
We're going through all the nodes, besides the first node. This means we already have an O(N) loop.
In each layer of the loop, we're considering all the edges that stem from a processed node to an unprocessed node.
However, in the worst-case, there are close to O(M) of these in each layer of the loop; not O(1) or anything less than O(M).
There's an easy way to prove this. If each edge were to only be visited a single time, then we can say that the algorithm runs in O(M). However, in your naive implementation of Dijkstra's algorithm, the same edge can be considered multiple times. In fact, asymptomatically speaking, O(N) times.
You can try creating a graph yourself, then dry-running the process of Dijkstra on paper. You'll notice that each edge can be considered up to N times: once in each layer of the O(N) loop.
In other words, the reason we can't say the program is faster than O(NM) is because there is no guarantee that each edge isn't processed N times, or less than O(log N) times, or less than O(sqrt N) times, etc. Therefore, the best upper bound we can give is N, although in practice it may be less than N by some sort of constant. However, we do not consider this constant in big-O time complexity.
However, your thought process may lead to a better implementation of Dijsktra. When considering the algorithm, you may realize that instead of considering every single edge that goes from processed to unprocessed vertices in every iteration of the main loop, we only have to consider every single edge that is adjacent to the current node we are on (I'm talking about something like this or this).
By implementing it like so, you can achieve a complexity of O(N^2).

Algorithmic complexity of group average clustering

I've been reading lately about various hierarchical clustering algorithms such as single-linkage clustering and group average clustering. In general, these algorithms don't tend to scale well. Naive implementations of most hierarchical clustering algorithms are O(N^3), but single-linkage clustering can be implemented in O(N^2) time.
It is also claimed that group-average clustering can be implemented in O(N^2 logN) time. This is what my question is about.
I simply do not see how this is possible.
Explanation after explanation, such as:
http://nlp.stanford.edu/IR-book/html/htmledition/time-complexity-of-hac-1.html
http://nlp.stanford.edu/IR-book/completelink.html#averagesection
https://en.wikipedia.org/wiki/UPGMA#Time_complexity
... are claiming that group average hierarchical clustering can be done in O(N^2 logN) time by using priority queues. But when I read the actual explanation or pseudo-code, it always appears to me that it is nothing better than O(N^3).
Essentially, the algorithm is as follows:
For an input sequence of size N:
Create a distance matrix of NxN #(this is O(N^2) time)
For each row in the distance matrix:
Create a priority queue (binary heap) of all distances in the row
Then:
For i in 0 to N-1:
Find the min element among all N priority queues # O(N)
Let k = the row index of the min element
For each element e in the kth row:
Merge the min element with it's nearest neighbor
Update the corresponding values in the distance matrix
Update the corresponding value in priority_queue[e]
So it's that last step that, to me, would seem to make this an O(N^3) algorithm. There's no way to "update" an arbitrary value in the priority queue without scanning the queue in O(N) time - assuming the priority queue is a binary heap. (A binary heap gives you constant access to the min element and log N insertion/deletion, but you can't simply find an element by value in better than O(N) time). And since we'd scan the priority queue for each row element, for each row, we get (O(N^3)).
The priority queue is sorted by a distance value - but the algorithm in question calls for deleting the element in the priority queue which corresponds to k, the row index in the distance matrix of the min element. Again, there's no way to find this element in the queue without an O(N) scan.
So, I assume I'm probably wrong since everyone else is saying otherwise. Can someone explain how this algorithm is somehow not O(N^3), but in fact, O(N^2 logN) ?
I think you are saying that the problem is that in order to update an entry in a heap you have to find it, and finding it takes time O(N). What you can do to get round this is to maintain an index that gives, for each item i, its location heapPos[i] in the heap. Every time you swap two items to restore the heap invariant you then need to modify two entries in heapPos[i] to keep the index correct, but this is just a constant factor on the work done in the heap.
If you store the positions in the heap (which adds another O(n) memory) you can update the heap without scanning, on the changed positions only. These updates are restricted to two paths on the heap (one removal, one update) and execute in O(log n). Alternatively, you could binary-search by the old priority, which will likely be in O(log n), too (but slower, above approach is O(1)).
So IMHO you can indeed implement these in O(n^2 log n). But the implementation will still use a lot (O(n^2)) of memory, anything of O(n^2) does not scale. You usually
run out of memory before you run out of time if you have O(n^2) memory...
Implementing these data structures is quite tricky. And when not done well, this may end up being slower than a theoretically-worse approach. For example Fibonacci heaps. They have nice properties on paper, but have too high constant costs to pay off.
No, because the distance matrix is symmetrical.
if the first entry in row 0 is to column 5, distance of 1, and that is lowest in the system, then the first entry in row 5 must be the complementary entry to column 0, with a distance of 1.
In fact you only need a half matrix.

Why does Dijkstra's algorithm use decrease-key?

Dijkstra's algorithm was taught to me was as follows
while pqueue is not empty:
distance, node = pqueue.delete_min()
if node has been visited:
continue
else:
mark node as visited
if node == target:
break
for each neighbor of node:
pqueue.insert(distance + distance_to_neighbor, neighbor)
But I've been doing some reading regarding the algorithm, and a lot of versions I see use decrease-key as opposed to insert.
Why is this, and what are the differences between the two approaches?
The reason for using decrease-key rather than reinserting nodes is to keep the number of nodes in the priority queue small, thus keeping the total number of priority queue dequeues small and the cost of each priority queue balance low.
In an implementation of Dijkstra's algorithm that reinserts nodes into the priority queue with their new priorities, one node is added to the priority queue for each of the m edges in the graph. This means that there are m enqueue operations and m dequeue operations on the priority queue, giving a total runtime of O(m Te + m Td), where Te is the time required to enqueue into the priority queue and Td is the time required to dequeue from the priority queue.
In an implementation of Dijkstra's algorithm that supports decrease-key, the priority queue holding the nodes begins with n nodes in it and on each step of the algorithm removes one node. This means that the total number of heap dequeues is n. Each node will have decrease-key called on it potentially once for each edge leading into it, so the total number of decrease-keys done is at most m. This gives a runtime of (n Te + n Td + m Tk), where Tk is the time required to call decrease-key.
So what effect does this have on the runtime? That depends on what priority queue you use. Here's a quick table that shows off different priority queues and the overall runtimes of the different Dijkstra's algorithm implementations:
Queue | T_e | T_d | T_k | w/o Dec-Key | w/Dec-Key
---------------+--------+--------+--------+-------------+---------------
Binary Heap |O(log N)|O(log N)|O(log N)| O(M log N) | O(M log N)
Binomial Heap |O(log N)|O(log N)|O(log N)| O(M log N) | O(M log N)
Fibonacci Heap | O(1) |O(log N)| O(1) | O(M log N) | O(M + N log N)
As you can see, with most types of priority queues, there really isn't a difference in the asymptotic runtime, and the decrease-key version isn't likely to do much better. However, if you use a Fibonacci heap implementation of the priority queue, then indeed Dijkstra's algorithm will be asymptotically more efficient when using decrease-key.
In short, using decrease-key, plus a good priority queue, can drop the asymptotic runtime of Dijkstra's beyond what's possible if you keep doing enqueues and dequeues.
Besides this point, some more advanced algorithms, such as Gabow's Shortest Paths Algorithm, use Dijkstra's algorithm as a subroutine and rely heavily on the decrease-key implementation. They use the fact that if you know the range of valid distances in advance, you can build a super efficient priority queue based on that fact.
Hope this helps!
In 2007, there was a paper that studied the differences in execution time between using the decrease-key version and the insert version. See http://www.cs.sunysb.edu/~rezaul/papers/TR-07-54.pdf
Their basic conclusion was not to use the decrease-key for most graphs. Especially for sparse graphs, the non-decrease key is significantly faster than the decrease-key version. See the paper for more details.
There are two ways to implement Dijkstra: one uses a heap that supports decrease-key and another a heap that doesn't support that.
They are both valid in general, but the latter is usually preferred.
In the following I'll use 'm' to denote the number of edges and 'n' to denote the number of vertices of our graph:
If you want the best possible worst-case complexity, you would go with a Fibonacci heap that supports decrease-key: you'll get a nice O(m + nlogn).
If you care about the average case, you could use a Binary heap as well: you'll get O(m + nlog(m/n)logn). Proof is here, pages 99/100. If the graph is dense (m >> n), both this one and the previous tend to O(m).
If you want to know what happens if you run them on real graphs, you could check this paper, as Mark Meketon suggested in his answer.
What the experiments results will show is that a "simpler" heap will give the best results in most cases.
In fact, among the implementations that use a decrease-key, Dijkstra performs better when using a simple Binary heap or a Pairing heap than when it uses a Fibonacci heap. This is because Fibonacci heaps involve larger constant factors and the actual number of decrease-key operations tends to be much smaller than what the worst case predicts.
For similar reasons, a heap that doesn't have to support a decrease-key operation, has even less constant factors and actually performs best. Especially if the graph is sparse.

How to implement Prim's algorithm with a Fibonacci heap?

I know Prim's algorithm and I know its implementation but always I skip a part that I want to ask now. It was written that Prim's algorithm implementation with Fibonacci heap is O(E + V log(V)) and my question is:
what is a Fibonacci heap in brief?
How is it implemented? And
How can you implement Prim's algorithm with a Fibonacci heap?
A Fibonacci heap is a fairly complex priority queue that has excellent amoritzed asymptotic behavior on all its operations - insertion, find-min, and decrease-key all run in O(1) amortized time, while delete and extract-min run in amortized O(lg n) time. If you want a good reference on the subject, I strongly recommend picking up a copy of "Introduction to Algorithms, 2nd Edition" by CLRS and reading the chapter on it. It's remarkably well-written and very illustrative. The original paper on Fibonacci heaps by Fredman and Tarjan is available online, and you might want to check it out. It's dense, but gives a good treatment of the material.
If you'd like to see an implementation of Fibonacci heaps and Prim's algorithm, I have to give a shameless plug for my own implementations:
My implementation of a Fibonacci heap.
My implementation of Prim's algorithm using a Fibonacci heap.
The comments in both of these implementations should provide a pretty good description of how they work; let me know if there's anything I can do to clarify!
Prim's algorithm selects the edge with the lowest weight between the group of vertexes already selected and the rest of the vertexes.
So to implement Prim's algorithm, you need a minimum heap. Each time you select an edge you add the new vertex to the group of vertexes you've already chosen, and all its adjacent edges go into the heap.
Then you choose the edge with the minimum value again from the heap.
So the time complexities we get are:
Fibonacci:
Choosing minimum edge = O(time of removing minimum) = O(log(E)) = O(log(V))
Inserting edges to heap = O(time of inserting item to heap) = 1
Min heap:
Choosing minimum edge = O(time of removing minimum from heap) = O(log(E)) = O(log(V))
Inserting edges to heap = O(time of inserting item to heap) = O(log(E)) = O(log(V))
(Remember that E =~ V^2 ... so log(E) == log(V^2) == 2Log(V) = O(log(V))
So, in total you have E inserts, and V minimum choosings (It's a tree in the end).
So in Min heap you'll get:
O(Vlog(V) + Elog(V)) == O(Elog(V))
And in Fibonacci heap you'll get:
O(Vlog(V) + E)
I implemented Dijkstra using Fibonacci heaps a few years ago, and the problem is pretty similar. Basically, the advantage of Fibonacci heaps is that it makes finding the minimum of a set a constant operation; so that's very appropriate for Prim and Dijkstra, where at each step you have to perform this operation.
Why it's good
The complexity of those algorithms using a binomial heap (which is the more "standard" way) is O(E * log V), because - roughly - you will try every edge (E), and for each of them you will either add the new vertex to your binomial heap (log V) or decrease its key (log V), and then have to find the minimum of your heap (another log V).
Instead, when you use a Fibonacci heap the cost of inserting a vertex or decreasing its key in your heap is constant so you only have a O(E) for that. BUT deleting a vertex is O(log V), so since in the end every vertex will be removed that adds a O(V * log V), for a total O(E + V * log V).
So if your graph is dense enough (eg E >> V), using a Fibonacci heap is better than a binomial heap.
How to
The idea is thus to use the Fibonacci heap to store all the vertices accessible from the subtree you already built, indexed by the weight of the smallest edge leading to it. If you understood the implementation or Prim's algorithm with using another data structure, there is no real difficulty in using a Fibonacci heap instead - just use the insert and deletemin methods of the heap as you would normally, and use the decreasekey method to update a vertex when you release an edge leading to it.
The only hard part is to implement the actual Fibonacci heap.
I can't give you all the implementation details here (that would take pages), but when I did mine I relied heavily on Introduction to algorithms (Cormen et al). If you don't have it yet but are interested in algorithms I highly recommend that you get a copy of it! It's language agnostic, and it provides detailed explanations about all the standards algorithms, as well as their proofs, and will definitely boost your knowledge and ability to use all of them, and design and prove new ones. This PDF (from the Wikipedia page you linked) provides some of the implementation details, but it's definitely not as clear as Introduction to algorithms.
I have a report and a presentation I wrote after doing that, that explain a bit how to proceed (for Dijkstra - see the end of the ppt for the Fibonacci heap functions in pseudo-code) but it's all in French... and my code is in Caml (and French) so I'm not sure if that helps!!! And if you can understand something of it please be indulgent, I was just starting programming so my coding skills were pretty poor at the time...

Running time for Dijkstra's algorithm on a priority queue implemented by sorted list/array

So I'm curious to know what the running time for the algorithm is on on priority queue implemented by a sorted list/array. I know for an unsorted list/array it is O((n^2+m)) where n is the number of vertices and m the number of edges. Thus that equates to O(n^2) time. But would it be faster if i used an sorted list/array...What would the running time be? I know extractmin would be constant time.
Well, Let's review what we need for dijkstra's algorithm(for future reference, usually vertices and edges are used as V and E, for example O(VlogE)):
Merging together all the sorted adjacency lists: O(E)
Extract Minimum : O(1)
Decrease Key : O(V)
Dijkstra uses O(V) extract minimum operations, and O(E) decrease key operations, therefore:
O(1)*O(V) = O(V)
O(E)*O(V) = O(EV) = O(V^2)
Taking the most asymptotically significant portion:
Eventual asymptotic runtime is O(V^2).
Can this be made better? Yes. Look into binary heaps, and better implementations of priority queues.
Edit: I actually made a mistake, now that I look at it again. E cannot be any higher than V^2, or in other words E = O(V^2).
Therefore, in the worst case scenario, the algorithm that we concluded runs in O(EV) is actually O(V^2 * V) == O(V^3)
I use SortedList
http://blog.devarchive.net/2013/03/fast-dijkstras-algorithm-inside-ms-sql.html
It is faster about 20-50 times than sorting List once per iteration

Resources