I am having trouble fully understanding this question:
Two O(n2) algorithms will always take the same amount of time for a given vale of n. True or false? Explain.
I think the answer is false, because from my understanding, I think that the asymptotic time complexity only measures the two algorithms running at O(n2) time, however one algorithm might take longer as perhaps it might have additional O(n) components to the algorithm. Like O(n2) vs (O(n2) + O(n)).
I am not sure if my logic is correct. Any help would be appreciated.
Yes, you are right. Big Oh notation depicts the upper bound of time complexity. There might some extra constant term c or smaller term of n like O(n) added to it which won't be considered for time complexity.
Moreover,
for i = 0 to n
for j = 0 to n
// some constant time operation
end
end
And
for i = 0 to n
for j = i to n
// some constant time operation
end
end
Both of these are O(n^2) asymptotically but won't take same time.
The concept of big Oh analysis is not to calculate the precise amount of time a program takes to execute, it's not about counting how many times a loop iterates. Rather it indicates the algorithm's growth rate with n.
The answer is correct but the explanation is lacking.
For one, the big O notation allows arbitrary constant factors. so both n2 and 100*n2 are in O(n2) but clearly the second is always larger.
Another reason is that the notation only gives an upper bound so even a runtime of n is in O(n2) so one of the algorithms may in fact be linear.
What would be the worst time complexity big O notation for the following pseudocode? (assuming the function call is an O(1)) I'm very new to big O notation so I'm unsure of an answer but I was thinking O(log(n)) because the while loop parameters multiplied by 2 each time or would that just be O(loglog(n))? Or am I wrong on both counts? Any input/help is appreciated, I'm trying to grasp the concept of big O notation for worst time complexity which I just started learning. Thanks!
i ← 1
while(i<n)
doSomething(...)
i ← i * 2
done
If i is doubling every time, then the number of times the loop will execute is the number of times you can double i before reaching n. Or to write it mathematically, if x is the number of times the loop will execute we have 2^x <= n. Solving for x gives x <= log_2(n). Therefore the number of times the loop will execute is O(log(n))
i is growing exponentially, thus loop will be done in logarithmic time, O(log(n))
O(log(n)) is correct when you want to state the time complexity of that algorithm in terms of the number n. However in computer science complexity is often stated in the size of the input, i.e. the number of bits. Then your algorithm would be linear, i.e. in O(k) where k is the input size.
Typically, other operations like addition are also said to be linear not logarithmic. A logarithmic complexity usually means that an algorithm does not have to consider the complete input. (E.g. binary search).
If this is part of an exercise or you want to discuss complexity of algorithms in a computer science context this difference is important.
Also, if one would want to be really pedantic: comparison on large integers is not a constant time operation, and if you are considering the usual integer types, the algorithm is basically constant time as it only needs up to 32 or 64 iterations.
for i = 0 to size(arr)
for o = i + 1 to size(arr)
do stuff here
What's the worst-time complexity of this? It's not N^2, because the second one decreases by one every i loop. It's not N, it should be bigger. N-1 + N-2 + N-3 + ... + N-N+1.
It is N ^ 2, since it's the product of two linear complexities.
(There's a reason asymptotic complexity is called asymptotic and not identical...)
See Wikipedia's explanation on the simplifications made.
Think of it like you are working with a n x n matrix. You are approximately working on half of the elements in the matrix, but O(n^2/2) is the same as O(n^2).
When you want to determine the complexity class of an algorithm, all you need is to find the fastest growing term in the complexity function of the algorithm. For example, if you have complexity function f(n)=n^2-10000*n+400, to find O(f(n)), you just have to find the "strongest" term in the function. Why? Because for n big enough, only that term dictates the behavior of the entire function. Having said that, it is easy to see that both f1(n)=n^2-n-4 and f2(n)=n^2 are in O(n^2). However, they, for the same input size n, don't run for the same amount of time.
In your algorithm, if n=size(arr), the do stuff here code will run f(n)=n+(n-1)+(n-2)+...+2+1 times. It is easy to see that f(n) represents a sum of an arithmetic series, which means f(n)=n*(n+1)/2, i.e. f(n)=0.5*n^2+0.5*n. If we assume that do stuff here is O(1), then your algorithm has O(n^2) complexity.
for i = 0 to size(arr)
I assumed that the loop ends when i becomes greater than size(arr), not equal to. However, if the latter is the case, than f(n)=0.5*n^2-0.5*n, and it is still in O(n^2). Remember that O(1),O(n),0(n^2),... are complexity classes, and that complexity functions of algorithms are functions that describe, for the input size n, how many steps there is in the algorithm.
It's n*(n-1)/2 which is equal to O(n^2).
I've been applying for jobs and every time I hear questions about algorithm time/space complexity, I cringe and stumble. No matter how much I read, my brain seems to be programmed to not get any of it, and I think the reason is down to the fact I have a very low mathematical background due to skipping school. This may not be the usual S.O question, potentially even be removed due to being fundamentally about the maths, but at least I'm hoping I'll figure out where to go next with this question.
I don't know why job people would go into that, so here's just a few examples. The whole "complexity" thing is just to provide an indication of how much time (or memory) the algorithm uses.
Now, if you have an array with values, accessing the value at a given index is O(1) -- constant. It doesn't matter how many elements are the in array, if you have an index you can get the element directly.
If, on the other hand, you are looking for a specific value, you'll have no choice but to look at every element (at least until you find the one, but that doesn't matter for the complexity thing). Thus, searching in a random array is O(n): the runtime corresponds to the number of elements.
If, on the other hand, you have sorted array then you can do a "binary search", which would be O(log n). "Log n" is twos-logarithm, which is basically the inverse of 2^n. For example, 2^10 is 2*2*2*2...*2 10 times = 1024, and log2(1024) is 10. Thus, algorithms with O(log n) are generally considered pretty good: to find an element in a sorted array using binary search, if the array has up to 1024 elements then the binary search will only have to look at 10 of them to find any value. For 1025-2048 elements it would be 11 peeks at most, for 2049-4096 it would 12 and so on. So, adding more elements will only slowly increase the runtime.
Of course, things can get a lot worse. A trivial sorting algorithm tends to be O(n**2), meaning that it needs 2^2 = 4 "operations" for an array with just 2 elements, 3^2 = 9 if the array has 3, 4^2 = 16 if the array has 4 elements and so on. Pretty bad actually, considering that an array with just 1000 elements would already require 1000*1000 = 1 million compares to sort. This is called exponential growth, and of course it can get even worse: O(n^3), O(n^4) etc. Increasingly bad.
A "good" sorting algorithm is O(n*log n). Assuming an array with 1024 elements, this would be 1024*10=10240 compares --- much better than the 1 million we had before.
Just take these O(...) as indicators for the runtime behavior (or memory footprint if applied to memory). I did plug in real numbers to you can see how the numbers change, but those are not important, and usually these complexities are worst case. Nonetheless, by just looking at the numbers, "constant time" is obviously best, exponential is always bad because runtime (or memory use) skyrockets really fast.
EDIT: also, you are not really interested in constant factors; you don't usually see "O(2n)". This would still be "O(n)" -- the runtime relates directly to the number of elements.
To analyze the time/space complexity of an algorithm - a high school knowledge should be fine. I studied this in Uni. during my first semester and I was just fine.
The fields of interests for the basics are:
basic calculus (understanding what a "limit" and asymptote are)
Series calculations (especially sum of arithmetic progression is often used)
Basics knowledge of combinatorics ("how many options there are to...?")
Basics of polynomial arithmetrics (p(x) + q(x) = ?)
The above is true for analyzing complexity of algorithms. Calculating complexity of problems is much deeper field, which is still in research - theory of complexity. This requires extensive knowledge on set theory, theory of computation, advanced calculus, linear algebra and much more.
While knowing something about calculus, summing series, and discrete mathematics are all Good Things, from your question and from my limited experience in industry, I'd doubt that your interviewers are expecting that level of understanding.
In practice, you can make useful big-O statements about time and space complexity without having to do much mathematical thinking. Here are the basics, which I'll talk about in terms of time complexity just make the language less abstract.
A big-O time complexity tells you how the worst-case running time of your algorithm scales with the size of its input. The actual numbers you get from the big-O function are an indication of the number of constant time operations your algorithm will perform on a given size of input.
A big-O function, therefore, simply counts the number of constant time operations your algorithm will perform.
A constant time operation is said to be O(1). [Note that any fixed length sequence of constant time operations is also O(1) since the sequence also takes a constant amount of time.]
O(k) = O(1), for any constant k.
If your algorithm performs several operations in series, you sum their costs.
O(f) + O(g) = O(f + g)
If your algorithm performs an operation multiple times, you multiply the cost of the operation by the number of times it is performed.
n * O(f) = O(n * f)
O(f) * O(f) * ... * O(f) = O(f^n), where there are n terms on the left hand side
A classic big-O function is log(n), which invariably corresponds to "the height of the balanced tree containing n items". You can get away with just knowing that sorting is O(n log(n)).
Finally, you only report the fastest growing term in a big-O function since, as the size of the input grows, this will dominate all the other terms. Any constant factors are also discarded, since we're only interested in the scaling properties of the result.
E.g., O(2(n^2) + n) = O(n^2).
Here are two examples.
Bubble Sorting n Items
Each traversal of the items sorts (at least) one item into place. We therefore need n traversals to sort all the items.
O(bubble-sort(n)) = n * O(traversal(n))
= O(n * traversal(n))
Each traversal of the items involves n - 1 adjacent compare-and-swap operations.
O(traversal(n)) = (n - 1) * O(compare-and-swap)
= O((n - 1) * O(compare-and-swap))
Compare-and-swap is a constant time operation.
O(compare-and-swap) = O(1)
Collecting our terms, we get:
O(bubble-sort(n)) = O(n * (n - 1) * 1)
= O(n^2 - n)
= O(n^2)
Merge Sorting n Items
Merge-sort works bottom-up, merging items into pairs, pairs into fours, fours into eights, and so on until the list is sorted. Call each such set of operations a "merge-traversal". There can be at most log_2(n) merge traversals since n = 2 ^ log_2(n) and at each level we are doubling the sizes of the sub-lists being merged. Therefore,
O(merge-sort(n)) = log_2(n) * O(merge-traversal(n))
= O(log_2(n) * merge-traversal(n))
Each merge-traversal traverses all the input data once. Each input item is the subject of at least one compare-and-select operation and each compare-and-select operation chooses one of a pair of items to "emit". Hence
O(merge-traversal(n)) = n * O(compare-and-select)
= O(n * compare-and-select)
Each compare-and-select operation takes constant time:
O(compare-and-select) = O(1)
Collecting terms, we get
O(merge-sort(n)) = O(log_2(n) * n * 1)
= O(n * log_2(n))
= O(n * log(n)), since change of log base is
multiplication by a constant.
Ta daaaa!
So given the following program:
Is the time complexity of this program O(0)? In other words, is 0 O(0)?
I thought answering this in a separate question would shed some light on this question.
EDIT: Lots of good answers here! We all agree that 0 is O(1). The question is, is 0 O(0) as well?
From Wikipedia:
A description of a function in terms of big O notation usually only provides an upper bound on the growth rate of the function.
From this description, since the empty algorithm requires 0 time to execute, it has an upper bound performance of O(0). This means, it's also O(1), which happens to be a larger upper bound.
Edit:
More formally from CLR (1ed, pg 26):
For a given function g(n), we denote O(g(n)) the set of functions
O(g(n)) = { f(n): there exist positive constants c and n0 such that 0 ≤ f(n) ≤ cg(n) for all n ≥ n0 }
The asymptotic time performance of the empty algorithm, executing in 0 time regardless of the input, is therefore a member of O(0).
Edit 2:
We all agree that 0 is O(1). The question is, is 0 O(0) as well?
Based on the definitions, I say yes.
Furthermore, I think there's a bit more significance to the question than many answers indicate. By itself the empty algorithm is probably meaningless. However, whenever a non-trivial algorithm is specified, the empty algorithm could be thought of as lying between consecutive steps of the algorithm being specified as well as before and after the algorithm steps. It's nice to know that "nothingness" does not impact the algorithm's asymptotic time performance.
Edit 3:
Adam Crume makes the following claim:
For any function f(x), f(x) is in O(f(x)).
Proof: let S be a subset of R and T be a subset of R* (the non-negative real numbers) and let f(x):S ->T and c ≥ 1. Then 0 ≤ f(x) ≤ f(x) which leads to 0 ≤ f(x) ≤ cf(x) for all x∈S. Therefore f(x) ∈ O(f(x)).
Specifically, if f(x) = 0 then f(x) ∈ O(0).
It takes the same amount of time to run regardless of the input, therefore it is O(1) by definition.
Several answers say that the complexity is O(1) because the time is a constant and the time is bounded by the product of some coefficient and 1. Well, it is true that the time is a constant and it is bounded that way, but that doesn't mean that the best answer is O(1).
Consider an algorithm that runs in linear time. It is ordinarily designated as O(n) but let's play devil's advocate. The time is bounded by the product of some coefficient and n^2. If we consider O(n^2) to be a set, the set of all algorithms whose complexity is small enough, then linear algorithms are in that set. But it doesn't mean that the best answer is O(n^2).
The empty algorithm is in O(n^2) and in O(n) and in O(1) and in O(0). I vote for O(0).
I have a very simple argument for the empty algorithm being O(0): For any function f(x), f(x) is in O(f(x)). Simply let f(x)=0, and we have that 0 (the runtime of the empty algorithm) is in O(0).
On a side note, I hate it when people write f(x) = O(g(x)), when it should be f(x) ∈ O(g(x)).
Big O is asymptotic notation. To use big O, you need a function - in other words, the expression must be parametrized by n, even if n is not used. It makes no sense to say that the number 5 is O(n), it's the constant function f(n) = 5 that is O(n).
So, to analyze time complexity in terms of big O you need a function of n. Your algorithm always makes arguably 0 steps, but without a varying parameter talking about asymptotic behaviour makes no sense. Assume that your algorithm is parametrized by n. Only now you may use asymptotic notation. It makes no sense to say that it is O(n2), or even O(1), if you don't specify what is n (or the variable hidden in O(1))!
As soon as you settle on the number of steps, it's a matter of the definition of big O: the function f(n) = 0 is O(0).
Since this is a low-level question it depends on the model of computation.
Under "idealistic" assumptions, it is possible you don't do anything.
But in Python, you cannot say def f(x):, but only def f(x): pass. If you assume that every instruction, even pass (NOP), takes time, then the complexity is f(n) = c for some constant c, and unless c != 0 you can only say that f is O(1), not O(0).
It's worth noting big O by itself does not have anything to do with algorithms. For example, you may say sin x = x + O(x3) when discussing Taylor expansion. Also, O(1) does not mean constant, it means bounded by constant.
All of the answers so far address the question as if there is a right and a wrong answer. But there isn't. The question is a matter of definition. Usually in complexity theory the time cost is an integer --- although that too is just a definition. You're free to say that the empty algorithm that quits immediately takes 0 time steps or 1 time step. It's an abstract question because time complexity is an abstract definition. In the real world, you don't even have time steps, you have continuous physical time; it may be true that one CPU has clock cycles, but a parallel computer could easily have asynchronoous clocks and in any case a clock cycle is extremely small.
That said, I would say that it's more reasonable to say that the halt operation takes 1 time step rather than that it takes 0 time steps. It does seem more realistic. For many situations it's arguably very conservative, because the overhead of initialization is typically far greater than executing one arithmetic or logical operation. Giving the empty algorithm 0 time steps would only be reasonable to model, for example, a function call that is deleted by an optimizing compiler that knows that the function won't do anything.
It should be O(1). The coefficient is always 1.
Consider:
If something grows like 5n, you don't say O(5n), you say O(n) [in other words, O(1n)]
If something grows like 7n^2, you don't say O(7n^2), you say O(n^2) [in other words, O(1n^2)]
Likewise you should say O(1), not O(some other constant)
There is no such thing as O(0). Even an oracle machine or a hypercomputer require the time for one operation, i.e. solve(the_goldbach_conjecture), ergo:
All machines, theoretical or real, finite or infinite produce algorithms with a minimum time complexity of O(1).
But then again, this code right here is O(0):
// Hello world!
:)
I would say it's O(1) by definition, but O(0) if you want to get technical about it: since O(k1g(n)) is equivalent to O(k2g(n)) for any constants k1 and k2, it follows that O(1 * 1) is equivalent to O(0 * 1), and therefore O(0) is equivalent to O(1).
However, the empty algorithm is not like, for example, the identity function, whose definition is something like "return your input". The empty algorithm is more like an empty statement, or whatever happens between two statements. Its definition is "do absolutely nothing with your input", presumably without even the implied overhead of simply having input.
Consequently, the complexity of the empty algorithm is unique in that O(0) has a complexity of zero times whatever function strikes your fancy, or simply zero. It follows that since the whole business is so wacky, and since O(0) doesn't already mean something useful, and since it's slightly ridiculous to even discuss such things, a reasonable special case for O(0) is something like this:
The complexity of the empty algorithm is O(0) in time and space. An algorithm with time complexity O(0) is equivalent to the empty algorithm.
So there you go.
Given the formal definition of Big O:
Let f(x) and g(x) be two functions defined over the set of real numbers. Then, we write:
f(x) = O(g(x)) as x approaches infinity iff there exists a real M and a real x0 so that:
|f(x)| <= M * |g(x)| for every x > x0
As I see it, if we substitute g(x) = 0 (in order to have a program with complexity O(0)), we must have:
|f(x)| <= 0, for every x > x0 (the constraint of existence of a real M and x0 is practically lifted here)
which can only be true when f(x) = 0.
So I would say that not only the empty program is O(0), but it is the only one for which that holds. Intuitively, this should've been true since O(1) encompasses all algorithms that require a constant number of steps regardless of the size of its task, including 0. It's essentially useless to talk about O(0); it's already in O(1). I suspect it's purely out of simplicity of definition that we use O(1), where it could as well be O(c) or something similar.
0 = O(f) for all function f, since 0 <= |f|, so it is also O(0).
Not only is this a perfectly sensible question, but it is important in certain situations involving amortized analysis, especially when "cost" means something other than "time" (for example, "atomic instructions").
Let's say there is a datastructure featuring multiple operation types, for which an amortized analysis is being conducted. It could well happen that one type of operation can always be funded fully using "coins" deposited during previous operations.
There is a simple example of this: the "multipop queue" described in Cormen, Leiserson, Rivest, Stein [CLRS09, 17.2, p. 457], and also on Wikipedia. Each time an item is pushed, a coin is put on the item, for a total amortized cost of 2. When (multi) pops occur, they can be fully paid for by taking one coin from each item popped, so the amortized cost of MULTIPOP(k) is O(0). To wit:
Note that the amortized cost of MULTIPOP is a constant (0)
...
Moreover, we can also charge MULTIPOP operations nothing. To pop the
first plate, we take the dollar of credit off the plate and use it to
pay the actual cost of a POP operation. To pop a second plate, we
again have a dollar of credit on the plate to pay for the POP
operation, and so on. Thus, we have always charged enough up front to
pay for MULTIPOP operations. In other words, since each plate on the
stack has 1 dollar of credit on it, and the stack always has a
nonnegative number of plates, we have ensured that the amount of
credit is always nonnegative.
Thus O(0) is an important "complexity class" for certain amortized operations.
O(1) means the algorithm's time complexity is always constant.
Let's say we have this algorithm (in C):
void doSomething(int[] n)
{
int x = n[0]; // This line is accessing an array position, so it is time consuming.
int y = n[1]; // Same here.
return x + y;
}
I am ignoring the fact that the array could have less than 2 positions, just to keep it simple.
If we count the 2 most expensive lines, we have a total time of 2.
2 = O(1), because:
2 <= c * 1, if c = 2, for every n > 1
If we have this code:
public void doNothing(){}
And we count it as having 0 expansive lines, there is no difference in saying it has O(0) O(1), or O(1000), because for every one of these functions, we can prove the same theorem.
Normally, if the algorithm takes a constant number of steps to complete, we say it has O(1) time complexity.
I guess this is just a convention, because you could use any constant number to represent the function inside the O().
No. It's O(c) by convention whenever you don't have dependence on input size, where c is any positive constant (typically 1 is used - O(1) = O(12.37)).