Shouldn't all recursive algorithm have worst case (Space) of O(inf) due to potential overflow?
Take Fibonacci algorithm What is the space complexity of a recursive fibonacci algorithm?
The answer is O(N) but what if the input is large and stack overflow occurs. Then the worst case is O(inf)
Short answer: No.
Slightly longer answer: As everyone mentioned in the comments, Big-O notation assumes infinite resources, so issues of implementation (in particular hardware related ones) have no bearing. It's only a description of the algorithm and how it hypothetically would scale if you give it arbitrarily large inputs.
That said, if you WERE to include hardware constraints in the definition, the answer would still be no, since Big-O is concerned with processor cycles, or iterations, or tasks -- that sort of measurement of doing something. If you get too deep in a recursive algorithm such that you run out of stack, you're gonna stop performing any algorithm-related operations whatsoever. It'll just stop and you won't be "doing anything" anymore. Nothing more to measure, and certainly not infinite.
And anyway, the point of Big-O is to be useful and help with understanding the theory of the algorithm. Saying "everything is Big-O of infinity", even if it were right, wouldn't accomplish any of that. It'd just be a loophole in the definition.
Related
BigO always checks the upper bound. So we can measure ..the way we write the code, so that there will be less time complexity and thus increase our code performance. But why do we use the lowerbound (omega) ? I did not understand the use of omega in real time. Can anybody please suggest me on this
It's a precision feature. It happens that it is usually easier to prove that an algorithm will take, say, O(n) operations to complete than proving that it will take at least O(n) operations (BTW, in this context operation means elementary computations such as the logical and arithmetic ones.)
By providing a lower bound, you are also giving an estimate of the best case scenario, as the big-O notation only provides an upper bound.
From a practical viewpoint, this has the benefit of telling that no matter what, any algorithm will require so many (elementary) steps (or more).
Note also, that it is also useful to have estimates of the average, the worst and the best cases, because these will shed more light on the complexity of the algorithm.
There are problems whose inherent complexity is known to be at least of some order (meaning there is a mathematical theorem proving the fact). So, no matter the algorithm, these problems cannot be solved with less that a certain number of calculations. This is also useful because lets you know whether a given algorithm is sub-optimal or matches the inherent complexity of the problem.
I was asked this question in an interview recently and was curious as to what others thought.
"When should you calculate Big O?"
Most sites/books talk about HOW to calc Big O but not actually when you should do it. I'm an entry level developer and I have minimal experience so I'm not sure if I'm thinking on the right track. My thinking is you would have a target Big O to work towards, develop the algorithm then calculate the Big O. Then try to refactor the algorithm for efficiency.
My question then becomes is this what actually happens in industry or am I far off?
"When should you calculate Big O?"
When you care about the Time Complexity of the algorithm.
When do I care?
When you need to make your algorithm to be able to scale, meaning that it's expected to have big datasets as input to your algorithm (e.g. number of points and number of dimensions in a nearest neighbor algorithm).
Most notably, when you want to compare algorithms!
You are asked to do a task, for which several algorithms can be applied to. Which one do you choose? You compare the Space, Time and development/maintenance complexities of them, and choose the one that best fits your needs.
Big O or asymptotic notations allow us to analyze an algorithm's running time by identifying its behavior as the input size for the algorithm increases.
So whenever you need to analyse your algorithm's behavior with respect to growth of the input, you will calculate this. Let me give you an example -
Suppose you need to query over 1 billion data. So you wrote a linear search algorithm. So is it okay? How would you know? You will calculate Big-o. It's O(n) for linear search. So in worst case it would execute 1 billion instruction to query. If your machine executes 10^7 instruction per second(let's assume), then it would take 100 seconds. So you see - you are getting an runtime analysis in terms of growth of the input.
When we are solving an algorithmic problem we want to test the algorithm irrespective of hardware where we are running the algorithm. So we have certain asymptotic notation using which we can define the time and space complexities of our algorithm.
Theta-Notation: Used for defining average case complexity as it bounds the function from top and bottom
Omega-Notation: Bounds the function from below. It is used for best-time complexity
Big-O Notation: This is important as it tells about worst-case complexity and it bounds the function from top.
Now I think the answer to Why BIG-O is calculated is that using it we can get a fair idea that how bad our algorithm can perform as the size of input increases. And If we can optimize our algorithm for worst case then average and best case will take care for themselves.
I assume that you want to ask "when should I calculate time complexity?", just to avoid technicalities about Theta, Omega and Big-O.
Right attitude is to guess it almost always. Notable exceptions include piece of code you want to run just once and you are sure that it will never receive bigger input.
The emphasis on guess is because it does not matter that much whether complexity is constant or logarithmic. There is also a little difference between O(n^2) and O(n^2 log n) or between O(n^3) and O(n^4). But there is a big difference between constant and linear.
The main goal of the guess, is the answer to the question: "What happens if I get 10 times larger input?". If complexity is constant, nothing happens (in theory at least). If complexity is linear, you will get 10 times larger running time. If complexity is quadratic or bigger, you start to have problems.
Secondary goal of the guess is the answer to question: 'What is the biggest input I can handle?". Again quadratic will get you up to 10000 at most. O(2^n) ends around 25.
This might sound scary and time consuming, but in practice, getting time complexity of the code is rather trivial, since most of the things are either constant, logarithmic or linear.
It represents the upper bound.
Big-oh is the most useful because represents the worst-case behavior. So, it guarantees that the program will terminate within a certain time period, it may stop earlier, but never later.
It gives the worst time complexity or maximum time require to execute the algorithm
This is a question from Introduction to Algorithms by Cormen et al, but this isn't a homework problem. Instead, it's self-study.
I have thought a lot and searched on Google. The answer that I can think of are:
Use another algorithm.
Give it best-case inputs
Use a better computer to run the algorithm
But I don't think these are correct. Changing the algorithm isn't the same as making an algorithm have better performance. Also using a better computer may increase the speed but the algorithm isn't better. This is a question in the beginning of the book so I think this is something simple that I am overlooking.
So how can we modify almost any algorithm to have a good best-case running time?
You can modify any algorithm to have a best case time complexity of O(n) by adding a special case, that if the input matches this special case - return a cached hard coded answer (or some other easily obtained answer).
For example, for any sort, you can make best case O(n) by checking if the array is already sorted - and if it is, return it as it is.
Note it does not impact average or worst cases (assuming they are not better then O(n)), and you basically improve the algorithm's best case time complexity.
Note: If the size of the input is bounded, the same optimization makes the best case O(1), because reading the input in this case is O(1).
If we could introduce an instruction for that very algorithm in the computation model of the system itself, we can just solve the problem in one instruction.
But as you might have already discovered that it is a highly unrealistic approach. Thus a generic method to modify any algorithm to have a best case running time is next to impossible. What we can do at max is to apply tweaks in the algorithm for general redundancies found in various problems.
Or you can go naive by taking the best case inputs. But again that isn't actually modifying the algorithm. In fact, introducing the algorithm in the computation system itself, instead of being highly unrealistic, isn't a modification in the algorithm either.
The ways we can modify the algorithm to have a best case running time are:
If the algorithm is at the point of its purpose/solution , For ex, for an increasing sort , it is already ascending order sorted and so on .
If we modify the algorithm such that we output and exit for its purpose only hence forcing multiple nested loops to be just one
We can sometimes use a randomized algorithm, that makes random choices, to allow a probabilistic analysis and thus improve the running time..
I think the only way for this problem is the input to the algorithm. Because the cases in time complexity analysis only depend on our input, how complex it is, how much times it tends to run the algorithm. on this analysis, we decide whether our case is best, average or worst.
So, our input will decide the running time for an algorithm in every case.
Or we can change our algorithm to improve for all cases(reducing the time complexity).
These are the ways we can achieve good best-case running time.
We can modify an algorithm for some special-case conditions, so if the input satisfies that condition, we can output the pre-computed answer. Generally, the best case running time is not a good measure for an algorithm. We need to know how the algorithm performes in worst case.
i just reached to this discussion while looking for an answer. what i think there is only one way to make any algorithm best case by having it a fixed input instead of varing input. if we have an fixed input always the cost and time complexity will always be O(1)
Is somebody aware of a natural program or algorithm that has a non-monotone worst-case behavior?
By non-monotone worst-case behavior I mean that there is a natural number n such that the worst-case runtime for inputs of size n+1 is less than the worst-case runtime for inputs of size n.
Of course, it is easy to construct a program with this behavior. It might even be the case that this happens for small n (like n = 1) in natural programs. But I'm interested in a useful algorithm that is non-monotone for large n.
Is somebody aware of a natural program or algorithm that has a
non-monotone worst-case behavior?
Please define "natural program or algorithm". The concept "algorithm" has a definition I'm aware of, and there are certainly algorithms (as you correctly admit) which have non-monotone worst-case runtime complexity. Is a program "natural" if it does no unecessary work or has minimal runtime complexity for the class of problem it solves? In that case, would you argue that BubbleSort isn't an algorithm? More importantly, I can define a problem the most efficient solution to which has non-monotone worst-case behavior. Would such a problem be "unnatural"? What is your definition of a "natural problem"?
Of course, it is easy to construct a program with this behavior.
Then what's the real question? Until you commit to a definition of natural/useful algorithms and problems, your question has no answer. Are you interested only in pre-existing algorithms which people already use in the real world? If so, you should state that, and the problem becomes one of searching the literature. Frankly, I cannot imagine a reasonable definition of "natural, useful algorithm" which would preclude many examples of algorithms with non-monotone runtime complexity...
But I'm interested in a useful algorithm that is non-monotone for
large n.
Please define "useful algorithm". The concept "algorithm" has a definition I'm aware of, and there are certainly algorithms (as you correctly admit) which have non-monotone worst-case runtime complexity. Is an algorithm "useful" if it correctly solves some problem? I can easily define a problem which can be solved by an algorithm with non-monotone runtime complexity.
Think about a binary search.
When implementing binary search you need to think about the case where the array segment which you're splitting is of odd length. At that point you have 2 choices:
1. Round up/down
2. Check both indexes and make a decision before continuing.
If you choose the first case (lets assume you round down). For odd length arrays where the number you're searching for is the one passed the middle point, you'll have an extra iteration to make.
If that odd array was added one more element it would have saved you that extra iteration.
If you went for the second case, then most executions of the algorithm with more odd iterations then even would require more comparisons then if it was used with an extra element.
Note that all this is very implementation dependent, and so there can't be a real answer without a real algorithm (and moreover a real implementation).
Also all this is based on the assuming you're talking about actual run-time diff and not asymptotic diff. If that's not the case, then the answer would be "no". There is no algorithms with non-monotonic worst case asymptotic running time. That would defy the concept of "worst case".
For instance, suppose I have an algorithm that's O(n) and an algorithm that's an amortized O(n). Is it fair to say that in strictly big oh terms, the non-amortized algorithm will always be as fast or faster than the amortized algorithm? Or is there ever any reason to prefer the amortized version (ignoring things like code simplicity or ease of implementation)?
Big O notation only tells you how your code scales, not how fast it will be for finite N.
The difference between amortized and non-amortized is important if you care about worst-case performance or latency (such as real-time or interactive systems). If you only care about average throughput, though, they're for all practical purposes the same. Amortized analysis is good enough even in some very performance-critical situations like scientific computing and large-scale data mining.
Or is there ever any reason to prefer the amortized version
Smaller constants.
The main difference between a O(n) algorithm and an amortized O(n) algorithm is that you don't know anything about the worst-case behavior of the amortized O(n) algorithm. In practice, that doesn't matter very often: If your algorithm is being run many times, then you can count on the law of averages to balance out a few bad cases, and if your algorithm isn't being run very many times, then it's unlikely that you'll ever hit the worst case at all.
The only cases where the word "amortized" should matter are cases where you can't accept occasional bad performance for some reason. For example, in GUI applications, you would happily give up some average-case performance in return for a guarantee that you'll never bog down and stop responding while the user is sitting there and getting bored. In such applications, you want to make sure that even the worst-case behavior is good for any code that could cause the GUI to stop responding.
Still, most of the time, you don't need to worry about amortized O(n) versus O(n), and you can worry instead about what the constant factors are (as others have already said).
Big O notation tells you about how your algorithm changes with growing input.
It is not a shortcut for profiling your code.
It is possible that a better algorithm is O(n^2) for all n in your program, because there is a constant in an O(n) that is larger.
So your choice of algorithms really depends on which algorithm is faster for your input size. I guess the answer to your question is to profile both algorithms in your program and then decide which is faster.
A classic example of the need for amortized algorithm would be std::vector for which insert is amortized O(1).
Some reasons to use amortized algorithms:
Much more efficient average case.
Easier Implementation.
No worst case guarantee algorithm exists.
Strictly speaking big-O isn't precise enough of a measure to so be able to say that an algorithm with O(n) is faster than one with a amortized O(n). Think about it. If you drop down a level of fidelity in your complexity analysis, the constant factors may be significantly different and make the amortized version faster.
Also, the impact of amortization tends to depend on use. For example, if you're using a hash table, the impact of the amortization will be largely dependent on your ratio of get to add operations. So if you add 1000 entries, then do a billion lookups, the fact that you had to rehash a couple times doesn't make much of a difference. But if you're constantly adding entries the cost of rehashing might add up.
That's why amortization is different than worst case. Big-O reflects the worst case, while amortization lets you say "one in every x will take a hit, and x is sufficiently large that it doesn't matter." Also, if you consider examples like insertion into a hash table, x grows according to some constant. So if you have a hash table that starts with 100 buckets and doubles every rehash, then the rehashes are asymptotically going to get farther and farther apart. In addition, the absolute worst case complexity for an algorithm with an amortized complexity is dependent on previous calls, while in measures like average case the calls are considered independent.