The great people at MyCodeSchool.com have this introductory video on YouTube, covering the basics of Big-O, Theta, and Omega notation.
The following definition of Big-O notation is provided:
O(g(n) ) := { f(n) : f(n) ≤ cg(n) }, for all n ≥ n0
My casual understanding of the equation is as follows:
Given a function f(), which takes as its input n, there exists another function g(), whose output is always greater than or equal to the output of f()--given two conditions:
g() is multiplied by some constant c
n is greater than some lower bound n0
Is my understanding correct?
Furthermore, the following specific example was provided to illustrate Big-O:
Given:
f(n) = 5n2 + 2n + 1
Because all of the following are true:
5n2 > 2n2, for all n
5n2 > 1n2, for all n
It follows that:
c = 5 + 2 + 1 = 8
Therefore, the video concludes, f(n) ≤ 8n2 for all n ≥ 1, and g(n) = 8n2
I think maybe the video concluded that n0 must be 1, because 1 is the only positive root of the equality 8n2 = 5n2 + 2n + 1 ( Negative one-third is also a root, but n is limited to whole numbers. So, no dice there. )
Is this the standard way of computing n0 for Big-O notation?
Take the largest powered factor in your polynomial
Multiply it by the sum of the coefficients in your time function
Set their product equal to your time function
Solve for zero
Reject all roots that are not in the set of whole numbers
Any help would be greatly appreciated. Thanks in advance.
Your understanding is mostly correct, but from your wording - "I think maybe the video concluded that n0 must be 1", I have to point out that it is also valid to take n0 to be 2, or 3 etc. In fact, any number greater than 1 will satisfy the required condition, there are actually infinitely many choices for the pair (c, n0)!
The important point to note is that the values of the constant c and n0 does not really matter, all we care is the existence a pair of constants (c, n0).
The Basics
Big-O notation describes the asymptotic behavior of a given function f, it essential describes the upper bound of f when the its input value is sufficiently large.
Formally, we say that f is big-O of another function g, i.e. f(x) = O(g(x)), if there exists a positive constant c and a constant n0 such that the following inequality holds:
f(n) ≤ c g(n), for all n ≥ n0
Note that the inequality captures the idea of upper bound: f is upper-bounded by a positive multiple of g. Moreover, the "for all" condition satisfies that the upper bound holds when the input n is sufficiently large (e.g. larger than n0).
How to Pick (c, n0)?
In order to prove f(x) = O(g(x)) for given functions f, g, all we need is to pick any pair of (c, n0) such that the inequality holds, and then we are done!
There is no standard way of finding (c, n0), just use whatever mathematical tools you find helpful. For example, you may fix n0, and then find c by using Calculus to compute the maximum value of f(x) / g(x) in the interval [n0, +∞).
In your case, it appears that you are trying to prove that a polynomial of degree d is big-O of xd, the proof of the following lemma gives a way to pick (c, n0):
Lemma
If f is a polynomial of degree d, then f(x) = O(xd).
Proof: We have f(x) = ad xd + ad-1 xd-1 + ... + a1 x + a0, for each coefficient ai, we have ai ≤ |ai| (absolute value of ai).
Take c = (|ad| + |ad-1| + ... + |a1| + |a0|) , and n0 = 1, then we have:
f(x) = ad xd + ad-1 xd-1 + ... + a1 x + a0
≤ |ad| xd + |ad-1| xd-1 + ... + |a1| x + |a0|
≤ (|ad| + |ad-1| + ... + |a1| + |a0|) xd
= c xd, for all x ≥ 1
Therefore we have f(x) = O(xd)
Related
In 3rd edition of CLRS specifically section 3.1 (page 47 in my book) they say
when a > 0, any linear function an + b is in O(n^2), which is easily verified by taking c = a + |b| and n0 = max(1,-b/a).
where n0 is the value such that when n >= n0 we could show that an + b <= cn^2 in a proof of the above.
I tried to verify this but I couldn't get very far :(
How did they choose these values of c and n0? I know that the only thing that matters is that there exists such a c and n0 such that the above is true to prove that an + b is O(n^2) but I wonder how did they choose specifically those values of c and n0? They don't seem arbitrary, its as if they applied some technique I have never seen before to obtain them.
Thanks.
Let's take the simple case where a and b are both positive. What the authors are trying to do is to create a value where the quadratic function dominates the linear function for n >= 1. That's it. They're just trying to create a general solution to show where the right parabola dominates any line.
So for n=1, the value of the linear function (i.e. l(n) = an + b) will be a+b when n=1. A dominating quadratic without any linear sub-functions (i.e. q(n) = c * n^2) would dominate the linear function, l(n) at n=1 if we choose c = a + b. So, q(n) = (a+b)n^2 dominates l(n) = an + b when n>=1, assuming a and b are both positive. You can check out examples for yourself for plotting 30x^2 and 10x + 20 on Densmos.
It's a bit trickier when b is negative, but the positive case is basically the point.
f(n) = 4 * 2n + 4n + 20n5
So, g(n) = 4n
Now our f(n) = O(g(n))
4 * 2n + 4n + 20n5 ≤ c*4n
How do we do this? I know how to do it for simple cases, but this one is far more complex. Would it go along the lines of removing the constant 4 and 20n5 to then have 2n + 4n ≤ c*4n?
Or would it be for any c > 4*2n + 20n5. It feels like a lame answer, so i'm going to assume i'm wrong. Would prefer if someone hinted at the idea of how to solve these problems rather than give me the answer, thank you.
Hint / preparations
In the context of asymptotic analysis and, in this case, Big-O notation specifically; generally when wanting to prove that inequalities such as
4 * 2^n + 4^n + 20n^5 ≤ c*4^n, (+)
for some constant c > 0,
for n larger than some constant n0; n > n0
holds, we approach the left hand side expression term by term. Since we're free to choose any constants c and n0 to show that (+) holds, we can always express the lower order terms as less or equal to (≤) the higher order term by making n sufficiently large, e.g., choosing the value of n0 as we see fit.
Solution (spoilers ahead!)
Below follows one way to show that (+) holds for some set of positive constants c and n0. Since you only asked for hints, I suggest you start with the section above, and return to this section in case you get stuck or want to verify the derivation you ended up using.
Term by term analysis (in terms of 4^n) of the left hand side expression of(+)` follows.
Term 4 * 2^n:
4 * 2^n = 4^n <=> (2*2)*2^n = (2^2)^n <=> 2^(n+2) = 2^(2n)
<=> n+2 = 2n => n = 2
=> 4 * 2^n ≤ 4^n for n ≥ 2 (i)
Term 4^n: Trivial
Term 20n^5:
for which n is 20 * n^5 = 4^n?
Graphical solution:
=> 20 * n^5 ≤ 4^n for n ≥~ 10.7 (choose 11) (ii)
Inserting inequalities (i) and (ii) in the lhs of (+) yields:
4 * 2^n + 4^n + 20n^5 ≤ 4^n + 4^n + 4^n = 3*4^n
^
for n>max(2,11)=11 <-- choice of n0 |
choice of c
Hence, we have showed that (+) holds for constants n0 = 11 and c=3. Naturally, the choice of these constants is not unique (in fact, if such constants exists, an infinite amount of them exists). Subsequently, the lhs of (+) is in O(4^n).
Now, I note that your title mentions Big-Θ (whereas your question covers only Big-O). For deriving that lhs of (+) is Θ(4^n), we need to find also a lower asymptotic bound on the lhs of (+) in terms of 4^n. Since n > 0, this is, in this case, quite trivial:
4 * 2^n + 4^n + 20n^5 ≥ c2*4^n ? for n > n0 ? (++)
=> 4 * 2^n + 4^n + 20n^5 ≥ 4^n, for n > 0
I.e., in addition to showing that (+) holds (which implies O(4^n)), we've shown that (++) holds for e.g. c2 = 1 and (re-use) n0 = 11, which implies that lhs of (+) is Θ(4^n).
One way to approach an asymptotic analysis of a function such as the left hand side of (+) would be to make use of the somewhat rigorous term-by-term analysis shown in this solution. In practice, however, we know that 4^n will quickly dominate the lower order terms, so we could've just chosen a somewhat large n0 (say 100) and tested, term by term, if the lower order terms could be replaced by the higher order term with less or equal to (≤) relation, given n>n0. Or, given in what context we need to make use of our asymptotic bounds, we could just glance at the function and, without rigour, directly state that the asymptotic behaviour of the function is naturally O(4^n), due to this being the dominant term. This latter method should, imo, only be used after one has grasped how to formally analyse the asymptotic behaviour of functions and algorithms in the context of Big-O/-Omega and -Theta notation.
The formal definition is
O(g(n)) = {f(n) | ∃c, n₀ : 0 ≤ f(n) ≤ c g(n), ∀ n ≥ n₀}
But when you want to check if f(x) ∈ O(g(n)), you can use the simpler
f(n)
lim sup ────── ≤ c
n → ∞ g(n)
In this case,
4*2ⁿ + 4ⁿ + 20n⁵
lim sup ────────────────── = 1
n → ∞ 4ⁿ
So yes, we can choose for example c = 1.
I'm trying to prove that this is correct for any function f and g with domain and co-domain N. I have seen it proven using limits, but apparently you can also prove it without them.
Essentially what I'm trying to prove is "If f(n) doesn't have a big-O of g(n) then g(n) must have a big-O of f(n). What I'm having trouble is trying to understand what "f doesn't have a big-O of g" means.
According to the formal definition of big-O, if f(n) = O(g(n)) then n>=N -> f(n) <= cg(n) for some N and a constant c. If f(n) != O(g(n)) I think that means there is no c that fulfills this inequality for all values of n. Yet I don't see what I can do to use that fact to prove g(n) = O(f(n)). That doesn't prove that a c' exists for g(n) <= c'f(n), which would successfully prove the question.
Not true. Let f(n) = 1 if n is odd and zero otherwise, and g(n) = 1 if n is even and zero otherwise.
To say that f is O(g) would say there is a constant C > 0 and N > 0 such that n > N implies f(n) <= C g(n). Let n = 2 * N + 1, so that n is odd. Then f(n) = 1 but g(n) = 0 so that f(n) <= C * g(n) is impossible. Thus, f is O(g) is not true.
Similarly, we can show that g is O(f) is not true.
First of all, your definition of big-O is a little bitt off. You say:
I think that means there is no c that fulfills this inequality for all values of n.
In actuality, you need to pick a value c that fulfills the inequality for any value of n.
Anyway, to answer the question:
I don't believe the statement in the question is true... Let's see if we can think of a counter-example, where f(n) ≠ O(g(n)) and g(n) ≠ O(f(n)).
note: I'm going to use n and x interchangeably, since it's easier for me to think that way.
We'd have to come up with two functions that continually cross each other as they go towards infinity. Not only that, but they'd have to continue to cross each other regardless of the constant c that we multibly them by.
So that leaves me thinking that the functions will have to alternate between two different time complexities.
Let's look at a function that alternates between y = x and y = x^2:
f(x) = .2 (x * sin(x) + x^2 * (1 - sin(x)) )
Now, if we create a similar function with a slightly offset oscillation:
g(x) = .2 (x * cos(x) + x^2 * (1 - cos(x)) )
Then these two functions will continue to cross each others' paths out to infinity.
For any number N that you select, no matter how high, there will be an x1 greater than N such that f(x) = x^2 and g(x) = x. Similarly, there will be an x2 such that g(x) = x^2 and f(x) = x.
At these points, you won't be able to choose any c1 or c2 that will ensure that f(x) < c1 * g(x) or that g(x) < c2 * f(x).
In conclusion, f(n) ≠ O(g(n)) does not imply g(n) = O(f(n)).
I am trying to get the correct Big-O of the following code snippet:
s = 0
for x in seq:
for y in seq:
s += x*y
for z in seq:
for w in seq:
s += x-w
According to the book I got this example from (Python Algorithms), they explain it like this:
The z-loop is run for a linear number of iterations, and
it contains a linear loop, so the total complexity there is quadratic, or Θ(n2). The y-loop is clearly Θ(n).
This means that the code block inside the x-loop is Θ(n + n2). This entire block is executed for each
round of the x-loop, which is run n times. We use our multiplication rule and get Θ(n(n + n2)) = Θ(n2 + n3)
= Θ(n3), that is, cubic.
What I don't understand is: how could O(n(n+n2)) become O(n3). Is the math correct?
The math being done here is as follows. When you say O(n(n + n2)), that's equivalent to saying O(n2 + n3) by simply distributing the n throughout the product.
The reason that O(n2 + n3) = O(n3) follows from the formal definition of big-O notation, which is as follows:
A function f(n) = O(g(n)) iff there exists constants n0 and c such that for any n ≥ n0, |f(n)| ≤ c|g(n)|.
Informally, this says that as n gets arbitrary large, f(n) is bounded from above by a constant multiple of g(n).
To formally prove that n2 + n3 is O(n3), consider any n ≥ 1. Then we have that
n2 + n3 ≤ n3 + n3 = 2n3
So we have that n2 + n3 = O(n3), with n0 = 1 and c = 2. Consequently, we have that
O(n(n + n2)) = O(n2 + n3) = O(n3).
To be truly formal about this, we would need to show that if f(n) = O(g(n)) and g(n) = O(h(n)), then f(n) = O(h(n)). Let's walk through a proof of this. If f(n) = O(g(n)), there are constants n0 and c such that for n ≥ n0, |f(n)| ≤ c|g(n)|. Similarly, since g(n) = O(h(n)), there are constants n'0, c' such that for n ≥ n'0, g(n) ≤ c'|h(n)|. So this means that for any n ≥ max(c, c'), we have that
|f(n)| ≤ c|g(n)| ≤ c|c'h(n)| = c x c' |h(n)|
And so f(n) = O(h(n)).
To be a bit more precise - in the case of the algorithm described here, the authors are saying that the runtime is Θ(n3), which is a stronger result than saying that the runtime is O(n3). Θ notation indicates a tight asymptotic bound, meaning that the runtime grows at the same rate as n3, not just that it is bounded from above by some multiple of n3. To prove this, you would also need to show that n3 is O(n2 + n3). I'll leave this as an exercise to the reader. :-)
More generally, if you have any polynomial of order k, that polynomial is O(nk) using a similar argument. To see this, let P(n) = ∑i=0k(aini). Then, for any n ≥ 1, we have that
∑i=0k(aini) ≤ ∑i=0k(aink) = (∑i=0k(ai))nk
so P(n) = O(nk).
Hope this helps!
n(n+n2) == n2 + n3
Big-O notation only cares about the dominant term as n goes to infinity, so the whole algorithm is thought of as Θ(n3).
O(n(n+n^2)) = O(n^2 + n^3)
Since the n^3 term dominates the n^2 term, the n^2 term is negligible and thus it is O(n^3).
The y loop can be discounted because of the z loop (O(n) + O(n^2) -> O(n^2))
Forget the arithmetic.
Then you're left with three nested loops that all iterate over the full length of 'seq', so it's O(n^3)
I'm starting to learn about Big-Oh notation.
What is an easy way for finding C and N0 for a given function?
Say, for example:
(n+1)5, or n5+5n4+10n2+5n+1
I know the formal definition for Big-Oh is:
Let f(n) and g(n) be functions mapping
nonnegative integers to real numbers.
We say that f(n) is O(g(n)) if there
is a real constant c > 0 and an
integer constant N0 >= 1
such that f(n) <= cg(n) for every integer N > N0.
My question is, what is a good, sure-fire method for picking values for c and N0?
For the given polynomial above (n+1)5, I have to show that it is O(n5). So, how should I pick my c and N0 so that I can make the above definition true without guessing?
You can pick a constant c by adding the coefficients of each term in your polynomial. Since
| n5 + 5n4 + 0n3 + 10n2 + 5n1 + 1n0 | <= | n5 + 5n5 + 0n5 + 10n5 + 5n5 + 1n5 |
and you can simplify both sides to get
| n5 + 5n4 + 10n2 + 5n + 1 | <= | 22n5 |
So c = 22, and this will always hold true for any n >= 1.
It's almost always possible to find a lower c by raising N0, but this method works, and you can do it in your head.
(The absolute value operations around the polynomials are to account for negative coefficients.)
Usually the proof is done without picking concrete C and N0. Instead of proving f(n) < C * g(n) you prove that f(n) / g(n) < C.
For example, to prove n3 + n is O(n3) you do the following:
(n3 + n) / n3 = 1 + (n / n3) = 1 + (1 / n2) < 2 for any n >= 1. Here you can pick any C >= 2 with N0 = 1.
You can check what the lim abs(f(n)/g(n)) is when n->+infitity and that would give you the constant (g(n) is n^5 in your example, f(n) is (n+1)^5).
Note that the meaning of Big-O for x->+infinity is that if f(x) = O(g(x)), then f(x) "grows no faster than g(x)", so you just need to prove that lim abs(f(x)/g(x)) exists and is less than +infinity.
It's going to depend greatly on the function you are considering. However, for a given class of functions, you may be able to come up with an algorithm.
For instance, polynomials: if you set C to any value greater than the leading coefficient of the polynomial, then you can solve for N0.
After you understand the magic there, you should also get that big-O is a notation. It means that you do not have to look for these coefficients in every problem you solve, once you made sure you understood what's going on behind these letters. You should just operate the symbols according to the notaion, according to its rules.
There's no easy generic rule to determine actual values of N and c. You should recall your calculus knowledge to solve it.
The definition to big-O is entangled with definition of the limit. It makes c satisfy:
c > lim |f(n)/g(n)|, given n approaches +infinity.
If the sequence is upper-bounded, it always has a limit. If it's not, well, then f is not O(g). After you have picked concrete c, you will have no problem finding appropriate N.