Just for clarification. If you have an algorithm that calls 3 different functions. Each of these functions has a runtime of logn. The runtime of the algorithm is bigO(log n) correct? The definition of bigO being f(n) = O(g(n)) means there are positive constants c and k, such that 0 ≤ f(n) ≤ cg(n) for all n ≥ k. The values of c and k must be fixed for the function f and must not depend on n. For this situation. we could look at c as 3 for the 3 functions and g(n) being logn?
It depends on how the algorithm calls these functions. If the algorithm looks like
function algorithm(input) {
f(input'); // size of input' = O(size of input)
g(input''); // size of input'' = (size of input)
h(input'''); // size of input''' = O(size of input)
}
then the running time is the sum of running times of the functions the algorithm calls. Thus if f, g, and h run in time O(log n) then the algorithm also runs in time O(log n).
Let's say that your function is f(n) and the 3 functions it calls are f_1(n), f_2(n) and f_3(n). Let also T(f(n)) be the running time of f(n).
If for any i, function f_i(n) has running time O(log(n)), then it means by the definition, that there exist c_i > 0 and n_i >= 0, such that for all n >= n_i, T(f_i(n)) <= c_i * log(n).
From the above fact, in order to proof that T(f(n)) is also O(log(n)), you just need to find the constants n0 >= 0, c > 0, such that for all n >= n0, T(f(n)) <= c * log(n).
It turns out that if you pick n0 = max(n_1, n_2, n_3), and c = 3 * max(c_1, c_2, c_3), the condition is fullfilled, so indeed T(f(n)) = O(log(n)). This is sufficient, because we know that the only thing f(n) does is that it calls f_1(n), f_2(n) and f_3(n), and each of these functions is called exactly once.
Related
I'm having tough time in understanding Big O time complexity.
Formal definition of Big O :
f(n) = O(g(n)) means there are positive constants c and k, such that 0 ≤ f(n) ≤ cg(n) for all n ≥ k. The values of c and k must be fixed for the function f and must not depend on n.
And worst time complexity of insertion sort is O(n^2).
I want to understand what is f(n), g(n), c and k here in case of insertion sort.
Explanation
It is not that easy to formalize an algorithm such that you can apply Big-O formally, it is a mathematical concept and does not easily translate to algorithms. Typically you would measure the amount of "computation steps" needed to perform the operation based on the size of the input.
So f is the function that measures how many computation steps the algorithm performs.
n is the size of the input, for example 5 for a list like [4, 2, 9, 8, 2].
g is the function you measure against, so g = n^2 if you check for O(n^2).
c and k heavily depend on the specific algorithm and how exactly f looks like.
Example
The biggest issue with formalizing an algorithm is that you can not really tell exactly how many computation steps are performed. Let's say we have the following Java code:
public static void printAllEven(int n) {
for (int i = 0; i < n; i++) {
if (i % 2 == 0) {
System.out.println(i);
}
}
}
How many steps does it perform? How deep should we go? What about for (int i = 0; i < count; i++)? Those are multiple statements which are executed during the loop. What about i % 2? Can we assume this is a "single operation"? On which level, one CPU cycle? One assembly line? What about the println(i), how many computation steps does it need, 1 or 5 or maybe 200?
This is not practical. We do not know the exact amount, we have to abstract and say it is a constant A, B and C amount of steps, which is okay since it runs in constant time.
After simplifying the analysis, we can say that we are effectively only interested in how often println(i) is called.
This leads to the observation that we call it precisely n / 2 times (since we have so many even numbers between 0 and n.
The exact formula for f using aboves constants would yield something like
n * A + n * B + n/2 * C
But since constants do not really play any role (they vanish in c), we could also just ignore this and simplify.
Now you are left with proving that n / 2 is in O(n^2), for example. By doing that, you will also get concrete numbers for c and k. Example:
n / 2 <= n <= 1 * n^2 // for all n >= 0
So by choosing c = 1 and k = 0 you have proven the claim. Other values for c and k work as well, example:
n / 2 <= 100 * n <= 5 * n^2 // for all n >= 20
Here we have choosen c = 5 and k = 20.
You could play the same game with the full formula as well and get something like
n * A + n * B + n/2 * C
<= n * (A + B + C)
= D * n
<= D * n^2 // for all n > 0
with c = D and k = 0.
As you see, it does not really play any role, the constants just vanish in c.
In case of insertion sort f(n) is the actual number of operations your processor will do to perform a sort. g(n)=n2. Minimal values for c and k will be implementation-defined, but they are not as important. The main idea this Big-O notation gives is that if you double the size of the array the time it takes for insertion sort to work will grow approximately by a factor of 4 (22). (For insertion sort it can be smaller, but Big-O only gives upper bound)
I have understood this roughly as given a function f(n), if I multiply it with some constant "K" & f(n) = O(g(n)) :: f(n) <= c.g(n) for some n>=n1 then if I make f(n) as Kf(n) then there must be some other constant c1 by which we can multiply g(n) and cap (put a higher limit) to Kf(n).
What I am finding difficult to understand is the proper mathematical explanation given in the book:
Let's assume we have a function f(n) which we know to be in O(g(n). This means, there exists an n0 and c, such that:
f(n) <= c * g(n) \forall n >= n0
Now, if we analyze K * f(n), it follows from the above definition that:
K * f(n) <= K * c * g(n) \forall n >= n0
We observe that K * c is again a constant. Hence, we can introduce another constant c' = K * c and write:
K * f(n) <= c' * g(n) \forall n >= n0
And this is exactly the big-O definition from above. Finally:
K * f(n) \in O(g(n))
It's just another constant.
it just means that no matter what the constant time is the asymptotic behavior stays the same. example:
int fact;
for (fact=1;n>1;n--) fact*=n;
is simple O(n) factorial with constant time c given by the time of single iteration of the loop and the multiplication fact*=n. Multiplying this by a constant means just that the c has changed for example:
int fact;
for (fact=1;n>1;n--)
if (bits(fact)+bits(n)<32)
fact*=n;
Now the c is bigger because I added condition into the loop which takes some time. But the O(n) stays. What changes is the runtime. On the other hand if I change the code like this:
int fact;
for (fact=1;n>1;n--)
if (bits(fact)+bits(n)<32) fact*=n;
else break;
Then I changed not only the constant time c but also the base asymptotic upper bound to O(1) because no matter how big n is this will stop when the result reached 32 bit limit which will be in constant time of steps for big enough n but this is not multiplying by a constant of coarse.
If a function body invokes 3 different functions, all of the order O(n), how do I calculate the order of the outer (containing) function? Yes, this is homework, and I've surprisingly failed to find a relevant example in the textbook, nor the slides of our recent lectures.
private void bigFunction(){
smallFunction1(); // O(n)
smallFunction2(); // O(n)
smallFunction3(); // O(n)
} // Now what does this result in?
My initial thought is O(n), but I want to be certain.
Yes, that's correct. The cost of doing any constant number of O(n) operations is O(n).
Specifically, O(n) × O(1) = O(n).
Hope this helps!
3 x O(n) = O(n) since we are trying to find time complexity, biggest complexity will be the answer, O(n) is the biggest in that algorithm.
You need to keep in mind the definition of big-oh:
A function f(x) is said to be O(g(n)) if there are numbers K and T
(which you can of course choose freely) such that for all x > T, f(x)
< K * g(x).
Of particular importance is the fact that you are free to select any K that fits the bill, not just the smallest K that does. It is this property that leads to g(n) always being shown as not having any constant factors: the following two scenarios are completely equivalent:
f(x) = x, g(n) = 2n, K = 1
f(x) = x, g(n) = 4n, K = 1/2
Since you can make g have any constant factor you like simply by selecting K appropriately, in practice we do not bother and always treat g as always having no constant factor.
At this point it should be clear that O(g(n)) + O(g(n)) is still O(g(n)), because for the sum you can simply choose "double the usual" value for K and still have the same form for g(n). Therefore the sum of any constant number of O(n) functions is still O(n).
The best way of being really sure of something like this is to construct a proof based on the definition, quoted in #Jon's answer:
A function f(x) is said to be O(g(n)) if there are numbers
K and T such that for all x > T, f(x) < K * g(x).
Let f_1(n) be the time for smallFunction1(), f_2(n) for smallFunction2(), and f_3(n) for smallFunction3(), all in a size n problem.
Because f_1(n) is O(n), there exist K_1 and T_1 such that, for all n > T_1, f_1(n) < K_1 * n.
Similarly, there exists K_2, T_2, K_3, and T_3 such that, for all n > T_2, f_2(n) < K_2 * n and for all n > T_3, f_3(n) < K_3 * n.
Let K equal K_1 + K_2 + K_3 and let T equal max(T_1, T_2, T_3). Then for all n > T, f_1(n) < K_1 * n, f_2(n) < K_2 * n, f_3(n) < K_3 * n.
The time to run the three functions consecutively, f_1(n) + f_2(n) + f_3(n), is less than K_1 * n + K_2 * n + K_3 * n = (K_1 + K_2 + K_3) * n = K * n, so the total time is O(n).
I know the definitions of both of them, but what is the reason sometimes I see O(1) and other times Θ(1) written in textbooks?
Thanks.
O(1) and Θ(1) aren't necessarily the same if you are talking about functions over real numbers. For example, consider the function f(n) = 1/n. This function is O(1) because for any n ≥ 1, f(n) ≤ 1. However, it is not Θ(1) for the following reason: one definition of f(n) = Θ(g(n)) is that the limit of |f(n) / g(n)| as n goes to infinity is some finite value L satisfying 0 < L. Plugging in f(n) = 1/n and g(n) = 1, we take the limit of |1/n| as n goes to infinity and get that it's 0. Therefore, f(n) ≠ Θ(1).
Hope this helps!
Big-O notation expresses an asymptotic upper bound, whereas Big-Theta notation additionally expresses an asymptotic lower bound. Often, the upper bound is what people are interested in, so they write O(something), even when Theta(something) would also be true. For example, if you wanted to count the number of things that are equal to x in an unsorted list, you might say that it can be done in linear time and is O(n), because what matters to you is that it won't take any longer than that. However, it would also be true that it's Omega(n) and therefore Theta(n), since you have to examine all of the elements in the list - it can't be done in sub-linear time.
UPDATE:
Formally:
f in O(g) iff there exists a c and an n0 such that for all n > n0, f(n) <= c * g(n).
f in Omega(g) iff there exists a c and an n0 such that for all n > n0, f(n) >= c * g(n).
f in Theta(g) iff f in O(g) and f in Omega(g), i.e. iff there exist a c1, a c2 and an n0 such that for all n > n0, c1 * g(n) <= f(n) <= c2 * g(n).
This question already has answers here:
What is a plain English explanation of "Big O" notation?
(43 answers)
Closed 9 years ago.
The definition in a book said
The "Big-Oh" Notation
Let f(n) and g(n) be functions mapping nonnegative integers to real numbers. We say that f(n) is O(g(n)) if there is a real constant c > 0 and an real constant n0 ≥ 1 such that
f(n) ≤cg(n), for n ≥ n0.
I couldn't able to understand terminologies used in formula and definition can somebody explain in plain English.
Basically, f(n) is O(g(n)) then g(n) is proportional to the worst-case scenario of f(x).
For example, binary search is O(log n) (or O(ln n), which is equivalent). Why?
(Binary search works like this: take the middle element, and compare to the target. If it's the one, you're done. If it's bigger than the target, throw out the second half of the list and repeat on the first half; if it's smaller than the target, throw out the first half and repeat the search on the second half.)
Because you need 1 operation to find something in a list that is 3 elements long; 2 operations when it's 7 elements long; 3 if it is 15 elements long. Thus, when number of elements n is (2^x - 1) for any x, the number of operations is x; turn it around, and you'd say for number of elements n, number of operations is log_2 n. And say that each operation lasts 2 seconds (say you're comparing stuff by hand), and the worst time to search is log_2 n * 2 seconds. log_2 n can be rewritten as ln n / ln 2, so the formula becomes:
worst search time(n) = (ln n / ln 2) * 2 seconds
= (2 seconds / ln 2) * ln n
Now, 2 seconds / ln 2 is a constant; let's call it c. Let's call "search time for n elements" f(n). And let's call ln n as g(n).
We said before, if n = 3, g(3) <= c * ln 3 (because c * ln 3 is worst search time, real search time is always less or equal to that; but we could always find it on our first try). If n = 7, g(7) <= c * ln 7; etc.
The bit about n0 is just a guard that says the complexity we calculate for the small n might be a deviation, an anomaly, an exception from the rule, and if we go with big enough data (i.e. n >= n0), the rule becomes obvious and inviolate. In this case, the rule works pretty much from the start, but some algorithms might have extra costs that throw off the calculation on small numbers.
Translation to "plain English": Imagine that f(n) are g(n) function that take a positive number or zero as input, and give a real number as output (no imaginary numbers).
Big-Oh allows us to compare two functions to see if one is bounded by the other. For example, an exponential function f(n) would not be bounded by a linear function g(n), so f(n) would not be O(g(n)).
We can say that f(n) is O(g(n)) if the following is possible: f(n) ≤ c * g(n) for n ≥ n0. If there is some way to solve the equation by plugging in for c and n0, then f(n) is O(g(n)).
For example (same as above), let f(n) = 2^n, g(n) = n. Is the following solvable: 2^n ≤ c * n for n ≥ n0? The answer is no. No matter what value is plugged into c, the left side will always be bigger than the right side as n approaches infinity. There is no way to make the left side smaller than the right side for all values n ≥ n0.
On the other hand, if f(n) = 2n, g(n) = n, then the condition is 2n ≤ c * n for n ≥ n0. This is solvable: c = 2, n0 = 0.
Let f(n) and g(n) be functions mapping nonnegative integers to real numbers.
Let f(n) and g(n) be functions where the values of n i.e. domain is 0 or positive integers, the values of f(n) and g(n) for those values of n may be real numbers.
We say that f(n) is O(g(n)) if there is a real constant c > 0 and an real constant n0 ≥ 1 such that:
f(n) ≤cg(n), for n ≥ n0.
f(n) = O(g(n)) if there exist positive constants c and n0 such that
0 <= f(n) <= cg(n) for all n >= n0. Actually , it means that f(n) is asymptotically less than or equal to g(n).
For example, consider f(n) = 3 * n^2 + 5. We can show that f(n) is O(n^2) by choosing c = 4 and n0 = 2. This is because for all values of n greater than 2:
3 * n^2 + 5 <= 4 * n^2
f(n) is not O(n), because whatever constant c and value n0 you choose, I can always find a value of n greater than n0 so that 3 * n^2 + 5 is greater than c * n.