I’m wondering whether there are any algorithms that use so much time that they must be represented using Knuth up-arrow notation.
Required: Use more than one up-arrow for time complexity.
Bonus points:
Have the algorithm be useful.
Have the algorithm be useful and optimized
Sister question on CS: (recommendation from #Amy)
https://cs.stackexchange.com/questions/94184/are-there-any-algorithms-that-run-in-2-↑-↑-n
Answering this to get this question off the unanswered queue.
As mentioned in the related question linked, a simple automaton-based algorithm for answering first-order queries in Presburger arithmetic (and some extensions) has worst-case running time about 2↑↑n, where n is the number of quantifier alternations in the query. It has both been implemented and provides useful results. Find more about it here
Related
My textbook is very poor at explaining how big-o works and gives little to no examples with no detail.
I have a few exercise questions I'm trying to attempt but thanks to the textbook
I don't understand how to tackle these questions.
Here is one:
determine whether each of these functions is O(x)
f(x)=x^2+x+1
and
determine whether each of these functions is O(x^2)
f(x)=xlogx
How do I go about solving these questions? From what I have gathered online and the textbook I find this very confusing..
Thanks in advance.
For the first one, x^2+x+1 is not O(x), as the first expression grows faster than the second no matter how large x gets. Typically, x^2+x+1 would be said to be O(x^2) ("quadratic"), as x^2 is the dominant term.
For the second one, xlogx is O(x^2) since the second expression grows at least as fast as the first. Example constraints would be c=1 and x>0. This is an overly-conservative expression though, and generally xlogx would be said to be O(xlogx) ("linearithmic"), its own complexity class.
The Wikipedia article on Big-O notation lists other common named complexities. While there are general methods to analyze a function and determine the its Big-O complexity, it's usually faster to just familiarize yourself with the common ones and recognize the most relevant one in an expression or algorithm. Usually you'll only encounter a few common complexity classes. In increasing order of complexity, these are:
Constant (1)
Logarithmic (logx)
Linear (x)
Linearithmic (or often just "n-log-n") (xlogx)
Poynomial (x^c for c>1)
Exponential (c^x for c>1)
I'm trying to find a source or two on the web that explain these in simple terms. Also, can this notion be used in a practical fashion to improve an algorithm? If so, how? Below is a brief explanation I got from a contact.
I dont know where you can find simple
explanation. But i try to explain you.
Algorithm complexity is a term, that
explain dependence between input
information and resourses that need to
process it. For example, if you need
to find max in array, you should
enumerate all elements and compare it
with your variable(e.g. max). Suppose
that there are N elements in array.
So, complexity of this algorithm is
O(N), because we enumerate all
elements for one time. If we enumerate
all elements for 2 times, complexity
will be O(N*N). Binary search has
complexity O(log2(N)), because its
decrease area of search by a half
during every step. Also, you can
figure out a space complexity of your
algorithm - dependence between space,
required by program, and amount of
input data.
It's not easy to say all things about complexity, but I think wiki has a good explanation on it and for startup is good, see:
Big O notation for introducing
this aspect (Also you can look at
teta and omega notations too).
Analysis of algorithm, to know
about complexity more.
And Computational Complexity,
which is a big theory in computer
science.
and about optimization you can look at web and wiki to find it, but with five line your friends give a good sample for introduction, but these are not one night effort for understanding their usage, calculation, and thousand of theory.
In all you can be familiar with them as needed, reading wiki, more advance reading books like Gary and Johnson or read Computation Complexity, a modern approach, but do not expect you know everything about them after that. Also you can see this lecture notes: http://www.cs.rutgers.edu/~allender/lecture.notes/.
As your friend hinted, this isn't a simple topic. But it is worth investing some time to learn. Check out this book, commonly used as a textbook in CS courses on algorithms.
The course reader used in Stanford's introductory programming classes has a great chapter on algorithmic analysis by legendary CS educator Eric Roberts. The whole text is online at this link, and Chapter 8 might be worth looking at.
You can watch Structure and Interpretation of computer programs. It's a nice MIT course.
Also, can this notion be used in a practical fashion to improve an algorithm? If so, how?
It's not so much used for improving an algorithm but evaluating the performance of algorithms and deciding on which algorithm you choose to use. For any given problem, you really want to avoid algorithms that has O(N!) or O(N^x) since they slow down dramatically when the size of N (your input) increases. What you want is O(N) or O(log(N)) or even better O(1).
O(1) is constant time which means the algorithm takes the same amount of time to execute for a million inputs as it does for one. O(N) is of course linear which means the time it takes to execute the algorithm increases in proportion to its input.
There are even some problems where any algorithm developed to solve them end up being O(N!). Basically no fast algorithm can be developed to solve the problem completely (this class of problems is known as NP-complete). Once you realize you're dealing with such a problem you can relax your requirements a bit and solve the problem imperfectly by "cheating". These cheats don't necessarily find the optimal solution but instead settle for good enough. My favorite cheats are genetic/evolutionary algorithms and rainbow tables.
Another example of how understanding algorithmic complexity changes how you think about programming is micro-optimizations. Or rather, not doing it. You often see newbies asking questions like is ++x faster than x++. Seasoned programmers mostly don't care and will usually reply the first rule of optimization is: don't.
The more useful answer should be that changing x++ to ++x does not in any way alter your algorithm complexity. The complexity of your algorithm has a much greater impact on the speed of your code than any form of micro-optimization. For example, it is much more productive for you to look at your code and reduce the number of deeply nested for loops than it is to worry about how your compiler turns your code to assembly.
Yet another example is how in games programming speeding up code counter-intuitively involve adding even more code instead of reducing code. The added code are in the form of filters (basically if..else statements) that decides which bit of data need further processing and which can be discarded. Form a micro-optimizer point of view adding code means more instructions for the CPU to execute. But in reality those filters reduce the problem space by discarding data and therefore run faster overall.
By all means, understand data structures, algorithms, and big-O.
Design code carefully and well, keeping it as simple as possible.
But that's not enough.
The key to optimizing is knowing how to find problems, after the code is written.
Here's an example.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 2 years ago.
Improve this question
I am trying to compare 2 algorithms. I thought I may try and write a proof for them. (My math sucks, so hence the question.)
Normally in our math lesson last year we would be given a question like <can't use symbols in here so left them out>.
Prove: (2r + 3) = n (n + 4)
Then I would do the needed 4 stages and get the answer at the end.
Where I am stuck is proving prims and Kruskals - how can I get these algorithms in to a form like the mathmatical one above, so I can proceed to prove?
Note: I am not asking people to answer it for me - just help me get it in to a form where I can have a go myself.
To prove the correctness of an algorithm, you typically have to show (a) that it terminates and (b) that its output satisfies the specification of what you're trying to do. These two proofs will be rather different from the algebraic proofs you mention in your question. The key concept you need is mathematical induction. (It's recursion for proofs.)
Let's take quicksort as an example.
To prove that quicksort always terminates, you would first show that it terminates for input of length 1. (This is trivially true.) Then show that if it terminates for input of length up to n, then it will terminate for input of length n+1. Thanks to induction, this is sufficient to prove that the algorithm terminates for all input.
To prove that quicksort is correct, you must convert the specification of comparison sorting to precise mathematical language. We want the output to be a permutation of the input such that if i ≤ j then ai ≤ aj. Proving that the output of quicksort is a permutation of the input is easy, since it starts with the input and just swaps elements. Proving the second property is a little trickier, but again you can use induction.
You don't give many details but there is a community of mathematicians (Mathematical Knowledge Management MKM) who have developed tools to support computer proofs of mathematics. See, for example:
http://imps.mcmaster.ca/
and the latest conference
http://www.orcca.on.ca/conferences/cicm09/mkm09/
Where i am stuck is proving prims and Kruskals - how can i get these algorithms in to a form like the mathmatical one above so i can proceed to prove
I don't think you can directly. Instead, prove that both generate a MST, then prove that any two MST are equal ( or equivalent, since you can have more than one MST for some graphs ). If both algorithms generate MSTs which are shown to be equivalent, then the algorithms are equivalent.
From my maths classes at Uni I (vaguely) remember proving Prims and Kruskals algorithms - and you don't attack it by writing it in a mathematical form. Instead, you take proven theories for Graphs and combine them e.g. http://en.wikipedia.org/wiki/Prim%27s_algorithm#Proof_of_correctness to build the proof.
If your looking to prove the complexity, then simply by the working of the algorithm it's O(n^2). There are some optimisations for the special case where the graph is sparse which can reduce this to O(nlogn).
Most of the times the proof depends on the problem you have in your hand. Simple argument can be suffice at times, at some other times you might need rigorous proof. I once used a corollary and proof of already proved theorem to justify my algorithm is right. But that is for a college project.
Maybe you want to try out a semi-automatic proof method. Just to go for something different ;) For example, if you have a Java specification of Prim's and Kruskal's algorithms, optimally building upon the same graph model, you can use the KeY Prover to prove the equivalence of the algorithm.
The crucial part is to formalize your proof obligation in Dynamic Logic (this is an extension of first-order logic with types and means of symbolic execution of Java programs). The formula to prove could match the following (sketchy) pattern:
\forall Graph g. \exists Tree t.
(<{KRUSKAL_CODE_HERE}>resultVar1=t) <-> (<{PRIM_CODE_HERE}>resultVar2=t)
This expresses that for all graphs, both algorithms terminate and the result is the same tree.
If you're lucky and your formula (and algorithm implementations) are right, then KeY can prove it automatically for you. If not, you might need to instantiate some quantified variables which makes it necessary to inspect the previous proof tree.
After having proven the thing with KeY, you can either be happy about having learned something or try to reconstruct a manual proof from the KeY proof - this can be a tedious task since KeY knows a lot of rules specific to Java which are not easy to comprehend. However, maybe you can do something like extracting an Herbrand disjunction from the terms that KeY used to instantiate existential quantifiers at the right-hand side of sequents in the proof.
Well, I think that KeY is an interesting tool and more people should get used to prove critical Java code using tools like that ;)
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
Is there any good resource (book, reference, web site, application...) which explains how to compute time complexity of an algorithm?
Because, it is hard to make the things concrete in my mind. Sometimes it is talking about an iteration has time complexity of lg n; and then according to the another loop it becomes n.lg n; and sometimes they use big omega notation to express it, sometimes big-o and etc...
These things are quite abstract to me. So, I need a resource which has good explanation and a lot of examples to make me see what's goin on.
Hopefully, I explained my problem clearly. I am quite sure that everyone who just started to study algorithms has also the same problem with me. So, maybe those experienced guys can also share their experience with us. Thanks...
I think the best book is Introduction to Algorithms by Cormen, Leiserson, Rivest and Stein.
Here is it on Amazon.
Guys, you're all recommending true complexity theory books -- Arora and Barak contains all sorts of things like PCP, Interactive Proofs, Quantum Computing and topics on Expander graphs -- things that most programmers/software developers have never heard of or will ever use. Papdimitriou is in the same category. Knuth is freaking impossible to read (and I was a CS/Math major) and gives zero intuition on how things work.
If you want a simple way to compute runtimes, and to get the flavour of the analysis, try the first chapter or so of Kleinberg and Tardos "Design and Analysis of Algorithms", that holds your hand through the fundamentals, and then you can work on much harder problems.
I read Christos Papadimitriou's Computational Complexity in university and really enjoyed it. It's not an easy matter, the book is long but it's well written (i.e., understandable and suitable for self-teaching) and it contains lots of useful knowledge, much more than just the "how do I figure out time complexity of algorithm x".
I agree that Introduction to Algorithms is a good book. For more detailed instructions on e.g. how to solve recurrence relations see Knuth's Concrete Mathematics. A good book on Computational Complexity itself is the one by Papadimitriou. But that last book might be a bit too thorough if you only want to calculate the complexity of given algorithms.
A short overview about big-O/-Omega notation:
If you can give an algorithm that solves a problem in time T(c*(n log n)) (c being a constant), than the time complexity of that problem is O(n log n). The big-O gets rid of the c, that is any constant factors not depending on the input size n. Big-O gives an upper bound on the running time, because you have shown (by giving an algorithm) that you can solve the problem in that amount of time.
If you can give a proof that any algorithm solving the problem takes at least time T(c*(n log n)) than the problem is in Omega(n log n). Big-Omega gives a lower bound on the problem. In most cases lower bounds are much harder to proof than upper bounds, because you have to show that any possible algorithm for the problem takes at least T(c*(n log n). Knowing a lower bound does not mean knowing an algorithm that reaches that lower bound.
If you have a lower bound, e.g. Omega(n log n), and an algorithm that solves the problem in that time (an upper bound), than the problem is in Theta(n log n). This means an "optimal" algorithm for this problem is known. Of course this notation hides the constant factor c which can be quite big (that's why I wrote optimal in quotes).
Note that when using these notations you are usually talking about the worst-case running time of an algorithm.
Computational complexity theory article in Wikipedia has a list of references, including a link to the book draft Computational Complexity: A Modern Approach, a textbook by Sanjeev Arora and Boaz Barak, Cambridge University Press.
The classic set of books on the subject is Knuth's Art of Computer Programming series. They're heavy on theory and formal proofs, so brush up on your calculus before you tackle them.
A course in Discrete Mathematics is sometimes given before Introduction to Algorithms.
Is there a master list of the Big-O notation for everything? Data structures, algorithms, operations performed on each, average-case, worst-case, etc.
Dictionary of Algorithms and Data Structures is a fairly comprehensive list, and includes complexity (Big-O) in the algorithms' descriptions. If you need more information, it'll be in one of the linked references, and there's always Wikipedia as a fallback.
The Cormen book is more about teaching you how to prove what Big-O would be for a given algorithm, rather than rote memorization of algorithm to its Big-O performance. The former is far more valuable than the latter, and requires an investment on your part.
Try "Introduction to Algorithms" by Cormen, Leisersen, and Rivest. If its not in there its probably not worth knowing.
In c++ the STL standards is defined by the Big-O characteristics of the algorithms as well as the space requirements. That way you could switch between competing implementations of STL and still know that your program had the same-ish runtime characteristics.
Particularily good STL implementations could even special case lists of particular types to be better than the standard-requirements.
It made it easy to pick the correct iterator or list type for a particular problem, because you could easily weigh between space consumption and speed.
Ofcourse Big-O is only a guide line as all constants are removed. If an algorithm runs in k*O(n), it would be classified as O(n), but if k is sufficiently high it could be worse than O(n^2) for some values of n and m.
Introduction to Algorithms, Second Edition, aka CLRS (Cormen, Leiserson, Rivest, Stein), is the closest thing I can think of.
If that fails, then try The Art of Computer Programming, by Knuth. If it's not in those, you probably need to do some real research.
To anyone, who is coming to this question from Google.
http://bigocheatsheet.com/