I'm interested in justification of the following line from the wikipedia article:
"This algorithm [extended euclidean algorithm] runs in time O(log(m)^2), assuming |a| < m, and is generally more efficient than exponentiation."
http://en.wikipedia.org/wiki/Modular_multiplicative_inverse
Why is this so? Can anyone explain this to me? I understand completely the algorithm and all the maths, it is just that i do not see how to determine the complexity of such algorithms. Any more general hints?
Also, additionally: Is log meant to be the natural logarithm (ln) or the one with base 2?
The popular Introduction to algorithms book (http://mitpress.mit.edu/books/introduction-algorithms) has got a whole chapter on proving algorithms complexity (however there's much more to the topic than in this book). You can read it if your generally interested in this matter.
You might also try to follow this paper's references: http://itee.uq.edu.au/~havas/cats03.pdf
Related
Please can someone tell me what the effectiveness of an algorithm relates to? I understand what the efficiency component entails
thanks
Effectiveness relates to the ability to produce the desired result.
Some tasks inherently do not have strict definitions - for example, machine translation between two human languages. Different algorithms exist to translate, say, from English to Spanish; their effectiveness is a measure of how good are the results that these algorithms produce. Their efficiency , on the other hand, measure how fast they are at producing the results, how much memory they use, how much disk space they need, etc.
This question suggests that you have read something which refers to the effectiveness of algorithms and have not understood the author's explanation of the term -- if the author has provided one. I don't think that there is a generally accepted interpretation of the term, I think it is one of those terms which falls under the Humpty-Dumpty rule 'a word means what I say it means'.
It might refer to an aspect of some algorithms which return only approximate solutions to problems. For example, we all know that the travelling salesman problem has NP time complexity, an actual algorithm which 'solves' the TSP might provide some bounds on the difference between the solutions it can find and an optimal solution which might take too long to find.
To compute the worst-case running time function of an algorithm what are the steps to be followed? Please some one guide me in that. I think these steps includes some mathematical proof's. If I am correct In which parts of mathematics areas I should be strong? (I guess mathematical Induction,functions, sets are enough)
Thanks
You can find good answers in the following books:
http://www.algorist.com/
"Art of computer programming", Knuth
I think mostly this is: good understanding of the algorithm, combinatorics and computational complexity theory - http://en.wikipedia.org/wiki/Computational_complexity_theory
To learn about computational complexity you need to know Calculus, Combinatorics, Set Theory, Summations amongst other maths topics.
A good book; though fairly theoretical is Introduction To Algorithms by Cormen et. al.
I'm trying to find a source or two on the web that explain these in simple terms. Also, can this notion be used in a practical fashion to improve an algorithm? If so, how? Below is a brief explanation I got from a contact.
I dont know where you can find simple
explanation. But i try to explain you.
Algorithm complexity is a term, that
explain dependence between input
information and resourses that need to
process it. For example, if you need
to find max in array, you should
enumerate all elements and compare it
with your variable(e.g. max). Suppose
that there are N elements in array.
So, complexity of this algorithm is
O(N), because we enumerate all
elements for one time. If we enumerate
all elements for 2 times, complexity
will be O(N*N). Binary search has
complexity O(log2(N)), because its
decrease area of search by a half
during every step. Also, you can
figure out a space complexity of your
algorithm - dependence between space,
required by program, and amount of
input data.
It's not easy to say all things about complexity, but I think wiki has a good explanation on it and for startup is good, see:
Big O notation for introducing
this aspect (Also you can look at
teta and omega notations too).
Analysis of algorithm, to know
about complexity more.
And Computational Complexity,
which is a big theory in computer
science.
and about optimization you can look at web and wiki to find it, but with five line your friends give a good sample for introduction, but these are not one night effort for understanding their usage, calculation, and thousand of theory.
In all you can be familiar with them as needed, reading wiki, more advance reading books like Gary and Johnson or read Computation Complexity, a modern approach, but do not expect you know everything about them after that. Also you can see this lecture notes: http://www.cs.rutgers.edu/~allender/lecture.notes/.
As your friend hinted, this isn't a simple topic. But it is worth investing some time to learn. Check out this book, commonly used as a textbook in CS courses on algorithms.
The course reader used in Stanford's introductory programming classes has a great chapter on algorithmic analysis by legendary CS educator Eric Roberts. The whole text is online at this link, and Chapter 8 might be worth looking at.
You can watch Structure and Interpretation of computer programs. It's a nice MIT course.
Also, can this notion be used in a practical fashion to improve an algorithm? If so, how?
It's not so much used for improving an algorithm but evaluating the performance of algorithms and deciding on which algorithm you choose to use. For any given problem, you really want to avoid algorithms that has O(N!) or O(N^x) since they slow down dramatically when the size of N (your input) increases. What you want is O(N) or O(log(N)) or even better O(1).
O(1) is constant time which means the algorithm takes the same amount of time to execute for a million inputs as it does for one. O(N) is of course linear which means the time it takes to execute the algorithm increases in proportion to its input.
There are even some problems where any algorithm developed to solve them end up being O(N!). Basically no fast algorithm can be developed to solve the problem completely (this class of problems is known as NP-complete). Once you realize you're dealing with such a problem you can relax your requirements a bit and solve the problem imperfectly by "cheating". These cheats don't necessarily find the optimal solution but instead settle for good enough. My favorite cheats are genetic/evolutionary algorithms and rainbow tables.
Another example of how understanding algorithmic complexity changes how you think about programming is micro-optimizations. Or rather, not doing it. You often see newbies asking questions like is ++x faster than x++. Seasoned programmers mostly don't care and will usually reply the first rule of optimization is: don't.
The more useful answer should be that changing x++ to ++x does not in any way alter your algorithm complexity. The complexity of your algorithm has a much greater impact on the speed of your code than any form of micro-optimization. For example, it is much more productive for you to look at your code and reduce the number of deeply nested for loops than it is to worry about how your compiler turns your code to assembly.
Yet another example is how in games programming speeding up code counter-intuitively involve adding even more code instead of reducing code. The added code are in the form of filters (basically if..else statements) that decides which bit of data need further processing and which can be discarded. Form a micro-optimizer point of view adding code means more instructions for the CPU to execute. But in reality those filters reduce the problem space by discarding data and therefore run faster overall.
By all means, understand data structures, algorithms, and big-O.
Design code carefully and well, keeping it as simple as possible.
But that's not enough.
The key to optimizing is knowing how to find problems, after the code is written.
Here's an example.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
Is there any good resource (book, reference, web site, application...) which explains how to compute time complexity of an algorithm?
Because, it is hard to make the things concrete in my mind. Sometimes it is talking about an iteration has time complexity of lg n; and then according to the another loop it becomes n.lg n; and sometimes they use big omega notation to express it, sometimes big-o and etc...
These things are quite abstract to me. So, I need a resource which has good explanation and a lot of examples to make me see what's goin on.
Hopefully, I explained my problem clearly. I am quite sure that everyone who just started to study algorithms has also the same problem with me. So, maybe those experienced guys can also share their experience with us. Thanks...
I think the best book is Introduction to Algorithms by Cormen, Leiserson, Rivest and Stein.
Here is it on Amazon.
Guys, you're all recommending true complexity theory books -- Arora and Barak contains all sorts of things like PCP, Interactive Proofs, Quantum Computing and topics on Expander graphs -- things that most programmers/software developers have never heard of or will ever use. Papdimitriou is in the same category. Knuth is freaking impossible to read (and I was a CS/Math major) and gives zero intuition on how things work.
If you want a simple way to compute runtimes, and to get the flavour of the analysis, try the first chapter or so of Kleinberg and Tardos "Design and Analysis of Algorithms", that holds your hand through the fundamentals, and then you can work on much harder problems.
I read Christos Papadimitriou's Computational Complexity in university and really enjoyed it. It's not an easy matter, the book is long but it's well written (i.e., understandable and suitable for self-teaching) and it contains lots of useful knowledge, much more than just the "how do I figure out time complexity of algorithm x".
I agree that Introduction to Algorithms is a good book. For more detailed instructions on e.g. how to solve recurrence relations see Knuth's Concrete Mathematics. A good book on Computational Complexity itself is the one by Papadimitriou. But that last book might be a bit too thorough if you only want to calculate the complexity of given algorithms.
A short overview about big-O/-Omega notation:
If you can give an algorithm that solves a problem in time T(c*(n log n)) (c being a constant), than the time complexity of that problem is O(n log n). The big-O gets rid of the c, that is any constant factors not depending on the input size n. Big-O gives an upper bound on the running time, because you have shown (by giving an algorithm) that you can solve the problem in that amount of time.
If you can give a proof that any algorithm solving the problem takes at least time T(c*(n log n)) than the problem is in Omega(n log n). Big-Omega gives a lower bound on the problem. In most cases lower bounds are much harder to proof than upper bounds, because you have to show that any possible algorithm for the problem takes at least T(c*(n log n). Knowing a lower bound does not mean knowing an algorithm that reaches that lower bound.
If you have a lower bound, e.g. Omega(n log n), and an algorithm that solves the problem in that time (an upper bound), than the problem is in Theta(n log n). This means an "optimal" algorithm for this problem is known. Of course this notation hides the constant factor c which can be quite big (that's why I wrote optimal in quotes).
Note that when using these notations you are usually talking about the worst-case running time of an algorithm.
Computational complexity theory article in Wikipedia has a list of references, including a link to the book draft Computational Complexity: A Modern Approach, a textbook by Sanjeev Arora and Boaz Barak, Cambridge University Press.
The classic set of books on the subject is Knuth's Art of Computer Programming series. They're heavy on theory and formal proofs, so brush up on your calculus before you tackle them.
A course in Discrete Mathematics is sometimes given before Introduction to Algorithms.
Is there a master list of the Big-O notation for everything? Data structures, algorithms, operations performed on each, average-case, worst-case, etc.
Dictionary of Algorithms and Data Structures is a fairly comprehensive list, and includes complexity (Big-O) in the algorithms' descriptions. If you need more information, it'll be in one of the linked references, and there's always Wikipedia as a fallback.
The Cormen book is more about teaching you how to prove what Big-O would be for a given algorithm, rather than rote memorization of algorithm to its Big-O performance. The former is far more valuable than the latter, and requires an investment on your part.
Try "Introduction to Algorithms" by Cormen, Leisersen, and Rivest. If its not in there its probably not worth knowing.
In c++ the STL standards is defined by the Big-O characteristics of the algorithms as well as the space requirements. That way you could switch between competing implementations of STL and still know that your program had the same-ish runtime characteristics.
Particularily good STL implementations could even special case lists of particular types to be better than the standard-requirements.
It made it easy to pick the correct iterator or list type for a particular problem, because you could easily weigh between space consumption and speed.
Ofcourse Big-O is only a guide line as all constants are removed. If an algorithm runs in k*O(n), it would be classified as O(n), but if k is sufficiently high it could be worse than O(n^2) for some values of n and m.
Introduction to Algorithms, Second Edition, aka CLRS (Cormen, Leiserson, Rivest, Stein), is the closest thing I can think of.
If that fails, then try The Art of Computer Programming, by Knuth. If it's not in those, you probably need to do some real research.
To anyone, who is coming to this question from Google.
http://bigocheatsheet.com/