I would like to prove that n^1000000 = O(1.000001^n) using the formal definition of big-O. I was wondering if I could do this using a proof by induction or a proof by contradiction. I would also like to do this without using limits.
The formal definition of big-O is: f(n) is O(g(n)) if and only if there exists a positive real number B and a non-negative real number b such that
|f(n)| <= B|g(n)| for all real number n > b
So we want to write the relationship between n^1000000 and 1.000001^n in such a way:
n^1000000 <= B * (1.000001^n) for all n > b
Notation: here we use log(n) for the natural log of n, so n = e^log(n)
n^1000000 = (e^log(n))^1000000 = e^(1000000 * log(n))
1.000001^n = (e^log(1.000001))^n = e^(log(1.000001) * n)
We know that n grows faster than log(n), so even though 1000000 > log(1.000001), log(1.000001) * n > 1000000 * log(n) if n is large enough.
That is the same as saying n^1000000 < 1.000001^n if n is larger than some number b (extremely large).
Therefore we can write
n^1000000 <= B * (1.000001^n) for all n > b
where B = 1 and b = some large number that can be calculated by solving 1.000001^b = b^1000000 and log(1.000001) * b > 1000000 * log(b).
Finally, we conclude that n^1000000 = O(1.000001^n).
You won't prove it because n^1000000 is a function and O(1.000001^n) is a set thus they can't be equal.
Quod erat demonstrandum
Related
I'm having tough time in understanding Big O time complexity.
Formal definition of Big O :
f(n) = O(g(n)) means there are positive constants c and k, such that 0 ≤ f(n) ≤ cg(n) for all n ≥ k. The values of c and k must be fixed for the function f and must not depend on n.
And worst time complexity of insertion sort is O(n^2).
I want to understand what is f(n), g(n), c and k here in case of insertion sort.
Explanation
It is not that easy to formalize an algorithm such that you can apply Big-O formally, it is a mathematical concept and does not easily translate to algorithms. Typically you would measure the amount of "computation steps" needed to perform the operation based on the size of the input.
So f is the function that measures how many computation steps the algorithm performs.
n is the size of the input, for example 5 for a list like [4, 2, 9, 8, 2].
g is the function you measure against, so g = n^2 if you check for O(n^2).
c and k heavily depend on the specific algorithm and how exactly f looks like.
Example
The biggest issue with formalizing an algorithm is that you can not really tell exactly how many computation steps are performed. Let's say we have the following Java code:
public static void printAllEven(int n) {
for (int i = 0; i < n; i++) {
if (i % 2 == 0) {
System.out.println(i);
}
}
}
How many steps does it perform? How deep should we go? What about for (int i = 0; i < count; i++)? Those are multiple statements which are executed during the loop. What about i % 2? Can we assume this is a "single operation"? On which level, one CPU cycle? One assembly line? What about the println(i), how many computation steps does it need, 1 or 5 or maybe 200?
This is not practical. We do not know the exact amount, we have to abstract and say it is a constant A, B and C amount of steps, which is okay since it runs in constant time.
After simplifying the analysis, we can say that we are effectively only interested in how often println(i) is called.
This leads to the observation that we call it precisely n / 2 times (since we have so many even numbers between 0 and n.
The exact formula for f using aboves constants would yield something like
n * A + n * B + n/2 * C
But since constants do not really play any role (they vanish in c), we could also just ignore this and simplify.
Now you are left with proving that n / 2 is in O(n^2), for example. By doing that, you will also get concrete numbers for c and k. Example:
n / 2 <= n <= 1 * n^2 // for all n >= 0
So by choosing c = 1 and k = 0 you have proven the claim. Other values for c and k work as well, example:
n / 2 <= 100 * n <= 5 * n^2 // for all n >= 20
Here we have choosen c = 5 and k = 20.
You could play the same game with the full formula as well and get something like
n * A + n * B + n/2 * C
<= n * (A + B + C)
= D * n
<= D * n^2 // for all n > 0
with c = D and k = 0.
As you see, it does not really play any role, the constants just vanish in c.
In case of insertion sort f(n) is the actual number of operations your processor will do to perform a sort. g(n)=n2. Minimal values for c and k will be implementation-defined, but they are not as important. The main idea this Big-O notation gives is that if you double the size of the array the time it takes for insertion sort to work will grow approximately by a factor of 4 (22). (For insertion sort it can be smaller, but Big-O only gives upper bound)
Prove or disprove the following claims:
Exist function f(n) so f(n-k) is not equal to Big-theta (f(n)). when k>=1 and is positive constant.
Is there any function which this claim is true?
I thought about f(n)=n! but I'm not sure that's is correct answer.
Moreover, if f(n)=n! is correct, how this claim can be proved?
Exist function so (f(n))^2=Big-O(f(n)) and f(n)=Big-omega (log(log(n))).
I think there is not function which make the claim to be true.
If this is correct - how it could be proved?
Correct for f(n) = n!. It suffices to show that for any fixed k >= 1, (n - k)! is not Omega(n!), as for any constant c > 0, it holds for all n large enough that c * n! > (n - k)!.
There is no f(n) such that both f(n)^2 = O(f(n)) and f(n) = Omega(log log n). The latter implies that for some constant c > 0 and all n large enough, f(n) > c log log n, and in particular f(n) > 1 for all n large enough. If we now assume that f(n)^2 = O(f(n)), then there exists a constant r > 0 so that for all n large enough, f(n)^2 < r * f(n), namely f(n) < r. But this implies that log log n < (r / c) for all n large enough, which is false for all n > e^(e^(r / c)) (where e is the basis of log).
A TA told me that this is true today but I was unable to verify this by googling. This is saying functions like log(n)^2, log(n)^3, ... , log(n)^m are all O(n).
Is this true?
Claim
The function f(n) = log(n)^m, for any natural number m > 2 (m ∈ ℕ+) is in
O(n).
I.e. there exists a set of positive constants c and n0 such that
the following holds:
log(n)^m < c · n, for all n > n0, { m > 2 | m ∈ ℕ+ } (+)
Proof
Assume that (+) does not hold, and denote this assumption as (*).
I.e., given (*), there exists no set of positive constants c and n0 such that (+) holds for any value of m > 2. Under this assumption, the following holds, that for all positive constants c and n0, there exists a n > n0 such that (thanks #Oriol):
log(n)^m ≥ c · n, { m > 2 | m ∈ ℕ+ } (++)
Now, if (++) holds, then the inequality in (++) will hold also after applying any monotonically increasing function to both sides of the inequality. One such function is, conveniently, the log function itself
Hence, under the assumption that (++) holds, then, for all positive constants c and n0, there exists a n > n0 such that the following holds
log(log(n)^m) ≥ log(c · n), { m > 2 | m ∈ ℕ+ }
m · log(log(n)) ≥ log(c · n), { m > 2 | m ∈ ℕ+ } (+++)
However, (+++) is obviously a contradiction: since log(n) dominates (w.r.t. growth) over log(log(n)),
we can—for any given value of m > 2—always find a set of constants c and n0 such that (+++) (and hence (++)) is violated for all n > n0.
Hence, assumption (*) is proven false by contradiction, and hence, (+) holds.
=> for f(n) = log(n)^m, for any finite integer m > 2, it holds that f ∈ O(n).
Yes. If the function it's f(n), it means m is a parameter and f does not depend on it. In fact, we have a different f_m function for each m.
f_m(n) = log(n)^m
Then it's easy. Given m ∈ ℕ, use L'Hôpital's rule repeatively
f_m(n) log(n)^m m * log(n)^(m-1)
limsup ──────── = limsup ────────── = limsup ────────────────── =
n→∞ n n→∞ n n→∞ n
m*(m-1) * log(n)^(m-2) m!
= limsup ──────────────────────── = … = limsup ──── = 0
n n→∞ n
Therefore, f_m ∈ O(n).
Of course, it would be different if we had f(m,n) = log(n)^m. For example, taking m=n,
f(n,n) log(n)^n
limsup ──────── = limsup ────────── = ∞
n→∞ n n→∞ n
Then f ∉ O(n)
In many ways it is more intuitive that for any positive integer m we have:
x^m = O(e^x)
This says that exponential growth dominates polynomial growth (which is why exponential time algorithms are bad news in computer programming).
Assuming that this is true, simply take x = log(n) and use the fact that then x tends to infinity if and only if n tends to infinity and that e^x and log(x) are inverses:
log(n)^m = O(e^log(n)) = O(n)
Finally, since for any natural number m, the root function n => n^(1/m) is increasing, we can rewrite the result as
log(n) = O(n^(1/m))
This way of writing it says that log(n) grows slower than any root (square, cube, etc.) of n, which more obviously corresponds to e^n growing faster than any power of n.
On Edit: the above showed that log(n)^m = O(n) followed from the more familiar x^m = O(e^x). To convert it to a more self-contained proof, we can show the later somewhat directly.
Start with the Taylor series for e^x:
e^x = 1 + x/1! + x^2/2! + x^3/3! + ... + x^n/n! + ...
This is known to converge for all real numbers x. If a positive integer m is given, let K = (m+1)!. Then, if x > K we have 1/x < 1/(m+1)!, hence
x^m = x^(m+1)/x < x^(m+1)/(m+1)! < e^x
which implies x^m = O(e^x). (The last inequality in the above is true since all terms in the expansion for e^x are strictly positive if x>0 and x^(m+1)/(m+1)! is just one of those terms.)
O( sqrt(n) ) = O(n) ?
We should find c and n0 verifying :
0 ( sqrt(n) ) < c*n ; c>0 and n>n0
How to find c and n0 ? or should i find another idea ?
Thanks
For n > 1, we have √n > 1, hence we have the following inequality:
√n < √n * √n = n, for any n > 1.
So we can take c = 1 and n0 = 2 to prove that √n = O(n).
Remark
Strictly speaking, you should avoid writing down something like O(√n) = O(n). Big-O notation is for describing the asymptotic upper bound of a function, but O(√n) is not a function.
O(√n) = O(n) is an abuse of notation, it really means the following:
If f is a function such that f(n) = O(√n), then f(n) = O(n).
In our case, if for any function f we have f(n) = O(√n), since √n < n for any n > 1, clearly we have f(n) < c * √n < c * n for any n > 1, and hence f(n) = O(n).
I can't understand the basic math behind algorithms. For example, here's a question:
If
f(n) = O(g(n))
is
f(n) * log(f(n)^c) = O(g(n) * log(g(n)))
?
How do I go about answering this question? From what I understand so far, f(n) = O(g(n)) only when g(n) <= c(g(n)) and c and n are non-negative. So I need to start plugging values into the above based on that, but how do I do that? Say if I chose c=5 and n=2, would I plug the values like so: f(2) * log(f(2)^5) = 5(g(2) * log(g(2))) Would that mean that the answer to the original question is false?
By f(n) = O(g(n)) , you mean
there exists a k such that f(n) <= k.g(n) for some n >= N_1. -----1
which implies log(f(n)^c) <= log(k^c) + log(g(n)^c) <= K*log(g(n)^c) for n >= N_2 and for K= max{log(k^c),2} -----2
which gives us the the required answer by multiplication of 1& 2
f(n) * log(f(n)^c) = O(g(n) * log(g(n))).
You can do it like that:
f(n) * log(f(n)^c) = c * f(n) * log(f(n)) = O(1) * O(g(n)) * log(O(g(n))) = O(1) * O(g(n)) * O(log(g(n))) = O(g(n) * log(g(n)).
So the answer to the question is true. All what you need here is properties of logarithm function.
There is one step that can be not clear here: why is log(O(g(n))) = O(log(g(n)))?
Proof: if f(n) = O(g(n)), there is a constant C such that for sufficiently large n
f(n) <= C * g(n). Thus, log(f(n)) <= log(C * g(n)) = log(g(n)) + log(C). We can use C2 = 2 and obtain log(f(n)) <= C2 * log(g(n)) for sufficiently large n.
You're given f(n) = O(g(n)) which means that beyond some point f(n) is at most a fixed positive multiple of g(n). You don't know where that point is, and you don't know the positive multiple. Still you know that
(⋆) f(n) ≤ k * g(n) for n > n₀ for some k > 0
even though k and n₀ are unknown. (I used k to avoid a name collision with c in the second part of the problem.)
Now you're asked to show that f(n) * log(f(n)^c) = O(g(n) * log(g(n))).
Logarithms turn multiplication into addition: log(x * y) = log(x) + log(y). That said, unsurprisingly, they turn repeated multiplication (which is exponentiation) into repeated addition (which is multiplication): log(x^y)= y * log(x).
First notice that log(f(n)^c) = c log(f(n)) because log(x^y) = y * log(x). So you may rewrite the problem: show c * f(n) * log(f(n)) = O(g(n) * log(g(n))).
What's more you can abandon the c on the left hand side: if something is at most a constant times something else (big-O), then any multiple of it is at most some other constant multiplied by the something else. Again you can rewrite the problem: show f(n) * log(f(n)) = O(g(n) * log(g(n))).
Now take the logarithm of (⋆):
(⋆) f(n) ≤ k * g(n) for n > n₀ for some k > 0
(⋆⋆) log(f(n)) ≤ log(k) + log(g(n)) for n > n₀ for some k > 0
The second only follows because log is an increasing function, so if numbers increase their logarithms increase --- taking the logarithm preserves order.
Now you need to multiply these two together to get something like the answer. Again this multiplication preserves order. If a ≤ A and b ≤ B then ab ≤ AB.
(⋆⋆⋆) f(n) * log(f(n)) ≤ k * g(n) * log(g(n)) + k * log(k) * g(n)
--- for n > n₀ for some k > 0
Now the left hand side is as in the problem, so you need to show the right hand side is at most some multiple of g(n) * log(g(n)) to finish up.
The first term on the right is exactly the sort of thing you need, but the second term is not a multiple of g(n) * log(g(n)), yet it increases the right hand side if log(k)>0, so you can't just throw it away: doing that would decrease the right hand side, and the left hand side might no longer be at most the right hand side.
(Of course if log(k)≤0 you can just throw the second term away which increases the right hand side and you're done: there's a multiple of g(n) * log(g(n)) there which is what you want.)
Instead what you need to do is increase k * log(k) * g(n) into another multiple of g(n) * log(g(n)). Increasing it means the left hand side is still at most the right hand side, and this makes the whole of the right hand side be a multiple of g(n) * log(g(n)) and the proof will be complete.
To do this, you need to multiply k * log(k) * g(n) by log(g(n)). Provided log(g(n)) ≥ 1 (again beyond some point), which seems reasonable in the theory of algorithms, this will make k * log(k) * g(n) bigger. (I could fuss here, but I won't!) So you may say that
(⋆⋆⋆) f(n) * log(f(n)) ≤ k * g(n) * log(g(n)) + k * log(k) * g(n)
--- for n > n₀ for some k > 0
But log(g(n)) ≥ 1 for n > n₁ and so multiplying the second term by log(g(n)) does not decrease it, so
f(n) * log(f(n)) ≤ k * g(n) * log(g(n)) + k * log(k) * g(n) * log(g(n))
--- for n > max(n₀,n₁) for some k > 0
and simplifying the right hand side
f(n) * log(f(n)) ≤ k(1+log(k)) * g(n) * log(g(n))
--- for n > max(n₀,n₁) for some k > 0
just gives a multiple of g(n) * log(g(n)) because k is a constant. i.e. take c = k(1+log(k)) to be the positive constant you need. (Earlier you eliminated the case where log(k) ≤ 0.) So
f(n) * log(f(n)) = O(g(n) * log(g(n)))