This is the question
I am taking a data structures class, i am kind of confused on how to go with this type of problem. Any leads will be helpful, thanks in advance.
Let me explain one of these and see if you can make an attempt at the rest.
f(n) is O( g(n) ) "function f of n" is ("big O") **Order** g(n)
if for some n (maybe not f(0) or f(1) or... but eventually for some n)
and for some **constant** (1, 2, 50, 80 million, something)
f(n) <= c * g(n)
So if we say some function f is "O(log n)" than means that starting at
some n that we pass into f(), and some number c then
f(n) <= c * log(n)
Lets take a really simple example:
function f ( n ) {
answer = 0;
for (i = 0; i < n; ++i) { // loops n times
answer += n+3; // 2 ops: add, add
answer /= (7*n); // 2 ops: mult, div
answer ^= 2; // 1 op: xor
} // 2 + 2 + 1 = 5
return answer;
}
So, we might say 'c' is 5, and g(n) is n (we clearly loop n times).
f (n) <= 5 * g(n)
f (n) <= 5 * n
f () is O(n)
basically what this is saying is the constant factors don't matter at all when n gets big enough. It makes almost no difference if f(n) is (5n) or (7n) or (27n) when we might compare it to other functions which could be (87log(n)) or (0.01n²).
\ n 10 1000 100000
f(n) \-----------------------------
7n | 70 7000 700000 O(n) grows linearly with prob size
87logn | ~200 ~600 ~1000 O(log n) grows slowly [good!]
.01n² | 10 10000 100000000 O(n²) grows fast [bad!]
Related
I'm having tough time in understanding Big O time complexity.
Formal definition of Big O :
f(n) = O(g(n)) means there are positive constants c and k, such that 0 ≤ f(n) ≤ cg(n) for all n ≥ k. The values of c and k must be fixed for the function f and must not depend on n.
And worst time complexity of insertion sort is O(n^2).
I want to understand what is f(n), g(n), c and k here in case of insertion sort.
Explanation
It is not that easy to formalize an algorithm such that you can apply Big-O formally, it is a mathematical concept and does not easily translate to algorithms. Typically you would measure the amount of "computation steps" needed to perform the operation based on the size of the input.
So f is the function that measures how many computation steps the algorithm performs.
n is the size of the input, for example 5 for a list like [4, 2, 9, 8, 2].
g is the function you measure against, so g = n^2 if you check for O(n^2).
c and k heavily depend on the specific algorithm and how exactly f looks like.
Example
The biggest issue with formalizing an algorithm is that you can not really tell exactly how many computation steps are performed. Let's say we have the following Java code:
public static void printAllEven(int n) {
for (int i = 0; i < n; i++) {
if (i % 2 == 0) {
System.out.println(i);
}
}
}
How many steps does it perform? How deep should we go? What about for (int i = 0; i < count; i++)? Those are multiple statements which are executed during the loop. What about i % 2? Can we assume this is a "single operation"? On which level, one CPU cycle? One assembly line? What about the println(i), how many computation steps does it need, 1 or 5 or maybe 200?
This is not practical. We do not know the exact amount, we have to abstract and say it is a constant A, B and C amount of steps, which is okay since it runs in constant time.
After simplifying the analysis, we can say that we are effectively only interested in how often println(i) is called.
This leads to the observation that we call it precisely n / 2 times (since we have so many even numbers between 0 and n.
The exact formula for f using aboves constants would yield something like
n * A + n * B + n/2 * C
But since constants do not really play any role (they vanish in c), we could also just ignore this and simplify.
Now you are left with proving that n / 2 is in O(n^2), for example. By doing that, you will also get concrete numbers for c and k. Example:
n / 2 <= n <= 1 * n^2 // for all n >= 0
So by choosing c = 1 and k = 0 you have proven the claim. Other values for c and k work as well, example:
n / 2 <= 100 * n <= 5 * n^2 // for all n >= 20
Here we have choosen c = 5 and k = 20.
You could play the same game with the full formula as well and get something like
n * A + n * B + n/2 * C
<= n * (A + B + C)
= D * n
<= D * n^2 // for all n > 0
with c = D and k = 0.
As you see, it does not really play any role, the constants just vanish in c.
In case of insertion sort f(n) is the actual number of operations your processor will do to perform a sort. g(n)=n2. Minimal values for c and k will be implementation-defined, but they are not as important. The main idea this Big-O notation gives is that if you double the size of the array the time it takes for insertion sort to work will grow approximately by a factor of 4 (22). (For insertion sort it can be smaller, but Big-O only gives upper bound)
I'm trying to learn how to find the big-theta bounds of various algorithms, but I'm having a hard time understanding how to do it, even after reading a number of questions here and lectures and textbooks on the subject. So for example
procedure for array a{
step=1
while(step <= n){
i = n
while(i >= step){
a[i]= a[i-step] + a[i]
i = i - 1}
step = step * 2}
}
I want to figure out the big-theta bound on the number of additions this goes through in terms of n, the number of indices in array a. I can see that the outer loop itself goes through log(n) iterations, but I can't figure out how to express what happens in the inner loop. Does anyone have an explanation or perhaps even a resource I might try consulting?
Big Theta notation asks us to find 2 constants, k1 and k2 such that our function f(n) is between k1*g(n) and k2*g(n) for sufficiently large n. In other words, can we find some other function g(n) that is at some point less than f(n) and at another point greater than f(n) (monotonically each way).
To prove Big-Theta, we need to find g(n) such that f(n) is O(g(n)) (upper bound), and f(n) is Big-Omega(g(n)) (lower bound).
Prove Big-O
In terms of Big-O notation (where f(n) <= g(n)*k), your algorithm, f(n), is O(log(n)*n), In this case g(n) = log(n) * n.
Let's prove this:
Find Inner Loop Executions
How many times does the outer loop execute? Track the "step" variable:
Let's say that n is 100:
1
2
4
8
16
32
64
128 (do not execute this loop)
That's 7 executions for an input of 100. We can equivalently say that it executes (log n) times (actually floor(log n) times, but log(n) is adequate).
Now let's look at the inner loop. Track the i variable, which starts at n and decrements until it is of size step each iteration. Therefore, the inner while loop will execute n - step times, for each value of step.
For example, when n = 100
100 - 1 = 99 iterations
100 - 2 = 98 iterations
100 - 4 = 96 iterations
100 - 8 = 92 iterations
100 - 16 = 84 iterations
100 - 32 = 68 iterations
100 - 64 = 36 iterations
So what's our total iteration count of the inner loop?
(n-1)
(n-1) + (n-2)
(n-1) + (n-2) + (n-4)
(n-1) + (n-2) + (n-4) + (n-8)
etc.
How is this thing growing? Well, because we know the outer loop will iterate log(n) times, we can formulate this thing as a summation:
Sum(from i=0 to log(n)) n-2^i
= log(n)*n - Sum(from i=0 to log(n)) 2^i
= log(n)*n - (2^0 + 2^1 + 2^2 + ... + 2^log(n))
= log(n)*n - ( (1-2^log(n) ) / (1-2) ) (actually 2^log(n+1) but close enough)
= log(n)*n + 1 - n
So now our goal is to show that:
log(n)*n + 1 - n = O(log(n)*n)
Clearly, log(n)*n is O(log(n)*n), but what about the 1-n?
1-n = O(log(n)*n)?
1-n <= k*log(n)*n, for some k?
Let k = 1
1-n <= log(n)*n?
Add n to both sides
1 <= n*log(n) + n? YES
So we've shown that f(n) is O(n*log(n)).
Prove Big-Omega
Now that we have an upper bound on f(n) using log(n) * n, let's try to get a lower bound on f(n) also using log(n) * n.
For a lower bound we use Big Omega notation. Big Omega looks for a function g(n)*k <= f(n) for some positive constant k.
k(n*log(n)) <= n*log(n) + 1 - n?
let k = 1/10
n*log(n)/10 <= n*log(n) + 1 - n?
(n*log(n) - 10n*log(n)) / 10 <= 1 - n?
-9n*log(n)/10 <= 1 - n? Multiply through by 10
-9n*log(n) <= 10 - 10n? Multiply through by -1 (flipping inequality)
9n*log(n) >= 10n - 10? Divide through by 9
n*log(n) >= 10n/9 - 10/9? Divide through by n
log(n) >= 10/9 - 10/9n ? Yes
Clearly, the quantity log(n) grows larger as (10/9 - 10/9n) tends towards 10/9. In fact for n = 1, 0 >= 10/9 - 10/9. 0 >= 0.
Prove Big-Theta
So now we've shown that f(n) is Big-Omega(n*log(n)). Combining this with the proof for f(n) is O(n*log(n)), and we've shown that f(n) is Big-Theta(n*log(n))! (the exclamation point is for excitement, not factorial...)
g(n) = n*log(n), and one valid set of constants is k1=1/10 (lower bound) and k2 = 1 (upper bound).
To prove big-O: there are floor(log2(n)) + 1 = O(log(n)) iterations through the outer loop, and the inner loop iterates O(n) times per, for a total of O(n * log(n)).
To prove big-Omega: there are floor(log2(n/2)) + 1 = Omega(log(n)) iterations through the outer loop during which step <= n/2. The inner loop iterates n + 1 - step times, which, for these outer iterations, is more than n/2 = Omega(n) per, for a total of Omega(n * log(n)).
Together, big-O and big-Omega prove big-Theta.
Simplifying the representation of your code like the following, we can translate your code into Sigma notation
for (step = 1; <= n; step = step * 2) {
for(i = n; i >= step; step = step - 1) {
}
}
Like this:
My homework involves Big O analysis and I think I've got the hang of it, but I'm not 100% sure. Would any of you lovely people mind taking a look and telling me if I'm on the right track?
The assignment is below. For questions 1 and 3, my analysis and answers are on the right, after the // marks. For question 2, my analysis and answers are below the algorithm type.
Thanks in advance for your help! :-)
1.For each of the following program fragments, give a Big-Oh analysis of the running time in terms of N:
(a) // Fragment (a)
for ( int i = 0, Sum = 0; i < N; i++ ) // N operations
for( int j = 0; j < N; j++ ) // N operations
Sum++; // Total: N^2 operations => O(N^2)
(b) // Fragment (b)
for( int i = 0, Sum = 0; i < N * N; i++ ) // N^2 operations
for( int j = 0; j < N; j ++ ) // N operations
Sum++; // Total: N^3 operations => O(N^3)
(c) // Fragment (c)
for( int i = 0, Sum = 0; i < N; i++ ) // N operations
for( int j = 0; j < i; j ++ ) // N-1 operations
Sum++; // Total: N(N-1) = N^2 – N operations => O(N^2)
(d) // Fragment (d)
for( int i = 0, Sum = 0; i < N; i++ ) // N operations
for( int j = 0; j < N * N; j++ ) // N^2 operations
for( int k = 0; k < j; k++ ) // N^2 operations
Sum++; // Total: N^5 operations => O(N^5)
2. An algorithm takes 0.5 milliseconds for input size 100. How long will it take for input size 500 if the running time is:
a. Linear
0.5 *5 = 2.5 milliseconds
b. O( N log N)
O (N log N) – treat the first N as a constant, so O (N log N) = O (log N)
Input size 100 = (log 100) + 1 = 2 + 1 = 3 operations
Input size 500 = (log 500) + 1= 2.7 + 1 = 3.7 ≈ 4 operations
Input size 100 runs in 0.5 milliseconds, so input size 500 takes 0.5 * (4/3) ≈ 0.67 milliseconds
c. Quadratic
Input size 100 in quadratic runs 100^2 operations = 10,000 operations
Input size 500 in quadratic runs 500^2 operations = 250,000 operations = 25 times as many
Input size of 100 runs in 0.5 milliseconds, so input size of 500 takes 25 * 0.5 = 12.5 milliseconds
d. Cubic
Input size 100 in quadratic runs 100^3 operations = 1,000,000 operations
Input size 500 in quadratic runs 500^3 operations = 125,000,000 operations = 125 times as many
Input size of 100 runs in 0.5 milliseconds, so input size of 500 takes 125 * 0.5 = 62.5 milliseconds
3. Find the Big-O for the following:
(a) f(x) = 2x^3 + x^2 log x // O(x^3)
(b) f(x) = (x^4 – 34x^3 + x^2 -20) // O(x^4)
(c) f(x) = x^3 – 1/log(x) // O(x^3)
4. Order the following functions by growth rate: (1 is slowest growth rate; 11 is fastest growth rate)
__6_ (a) N
__5_ (b) √N
__7_ (c) N^1.5
__9_ (d) N^2
__4_ (e) N log N
__2_ (f) 2/N
_11_ (g) 2^N
__3_ (h) 37
_10_ (i) N^3
__1_ (j) 1/ N^2
__8_ (k) N^2 /log N
* My logic in putting (j) and (f) as the slowest is that as N grows, 1/N^2 and 2/N decrease, so their growth rates are negative and therefore slower than the rest which have positive growth rates (or a 0 growth rate in the case of 37 (h)). Is that correct?
I looked at your questions 1 to 3 and its looks alright.
Follow these rules and check for yourself:
1) Multiplicative constants can be omitted,
Example 50n^2 simplifies to n^2
2) n^a dominates n^b if a>b
Example n^3 dominates n^2, so n^3 + n^2 + n ,simplifies to n3
3) Any exponential dominates any polynomial
Example 3^n dominates n^5
Example 2^n dominates n^2+5n+100
4) Any polynomial dominates any logarithm
Example n dominates (log n)3
As for Question 4 use below as a guide (from least to greatest):
Log2 n < n < n log2 n < n^2 < n^3 < 2^n
the answer for the (b) of the time calculation is wrong. you can not assume one of the n as constant.So nlogn becomes 1log1 which means log1 is 0. so 0.
so that answer is 100 log100 operations comparision with 500log500 ...
coming to the least to greatest.
b is 4 and a is 5. c,e,k are competition for the position 6 and 7 and 8.
1,2,3 positions given by you are correct.9,10,11 are correct.
i will check some analysis over 6,7,8 and let you know..
if you need any clarrification over my answer you can comment on that ..
#op Can you please tell me why you considered O(nlgn) = O(lg n)? As far as I understand your analysis for Q2 part b is actually the analysis of O(lg n) algorithms, to analyze nlgn algos you need to factor in that n in the left.
(a) Correct
(b) Correct
(c) Correctish. 0 + 1 + 2 + ... + (n - 1) = n(n - 1) / 2 = 0.5n^2 - 0.5n = O(n^2)
(d) Correct (there is a 1/2 in there like for (c), but the complexity is still O(N^5))
a. Correct
b. let K be the duration of one step.
K * (100 log 100) = 0.5, so K = 7.5 * 10^-4
K * (500 log 500) = 7.5 * 1-^-4 * 500 log 500 = 3.3737ms
Alternatively, (500 log 500) / (100 log 100) = 6.7474
When n = 500 it will be 6.7474 times slower, 6.7474 * 0.5ms = 3.3737ms
c. Correct
d. Correct
(a) Correct
(b) Correct
(c) Correct
__5_ (a) N
__4_ (b) √N
__7_ (c) N^1.5
__9_ (d) N^2
__6_ (e) N log N
__2_ (f) 2/N
_11_ (g) 2^N
__3_ (h) 37
_10_ (i) N^3
__1_ (j) 1 / N^2
__8_ (k) N^2 /log N
I agree with the positioning of (f) and (j). However, you should be aware that they don't occur out there 'in the wild' because every algorithm has at least one step, and therefore cannot beat O(1). See Are there any O(1/n) algorithms? for a more detailed explanation.
I am going over the Big-Oh notation, and I have a problem understanding the solution to this question:
Is 2n + 10 ≡ O(n)?
Can we find c and n0?
2n + 10 <= cn
(c-2)n >= 10
n >= 10/(c-2)
Pick c = 3 and n0 = 10
There is also a graph used in this example:
I am confused as to how c = 3 and how n0 = 10? Can somebody please enlighten me?
I would solve 2n + 10 <= cn differently.
2n + 10 <= cn
2 + 10/n <= c //divide by n
c >= 2 + 10/n
Clearly c has to be bigger than 2. Now take your favorite number bigger than 2. Uhm. c=2.718. This gives
2n + 10 <= 2.718*n
10 <= 0.718*n //subtract 2n
13.93 <= n
Thus, 2n + 10 <= c*n. If we take c=2.718 and n bigger than 15. (15 because it's bigger than the limit, 13.93, and I like 15.)
Any c bigger than 2 works, c=100000000000000000000000 is fine too (but, it costs much ink and paper to write down.)
It might have been easier to take c=3. That would give
2n + 10 <= 3*n
10 <= n //subtract 2n
This makes 3 the most logical / natural solution.
To say a function f(n) is O(n) means you can find c and n0 such that for all n >= n0, f(n) <= cn.
To verify this in your case: if n >= 10, then:
f(n) = 2n + 10
<= 3n // because 10 <= n
= cn
So f(n) <= cn for all n >= 10, so f(n) is an O(n) function.
Note that other values for c and n0 work; you just need to find one pair.
In Big-O notation you drop numbers added and multiplied.
And also use the largest power.
10*N^3+ 23*N^2 + 43*N + 154 = O(N^3)
let f(n) = 2n+10
as per the Big-Oh notation,
f(n)<= c*g(n) ----->eq1(let)
2n+10 <= 2n+n
2n+10 <= 3n for n>=10 //here the constant 10 is replaced by n ---->eq2
comparing eq1 with eq2
c=3 and g(n)=n for value of n=10
f(n)= O(g(n)) = O(n)
I'm reading a textbook right now for my Java III class. We're reading about Big-Oh and I'm a little confused by its formal definition.
Formal Definition: "A function f(n) is of order at most g(n) - that is, f(n) = O(g(n)) - if a positive real number c and positive integer N exist such that f(n) <= c g(n) for all n >= N. That is, c g(n) is an upper bound on f(n) when n is sufficiently large."
Ok, that makes sense. But hold on, keep reading...the book gave me this example:
"In segment 9.14, we said that an
algorithm that uses 5n + 3 operations
is O(n). We now can show that 5n + 3 =
O(n) by using the formal definition of
Big Oh.
When n >= 3, 5n + 3 <= 5n + n = 6n.
Thus, if we let f(n) = 5n + 3, g(n) =
n, c = 6, N = 3, we have shown that
f(n) <= 6 g(n) for n >= 3, or 5n + 3 =
O(n). That is, if an algorithm
requires time directly proportional to
5n + 3, it is O(n)."
Ok, this kind of makes sense to me. They're saying that if n = 3 or greater, 5n + 3 takes less time than if n was less than 3 - thus 5n + n = 6n - right? Makes sense, since if n was 2, 5n + 3 = 13 while 6n = 12 but when n is 3 or greater 5n + 3 will always be less than or equal to 6n.
Here's where I get confused. They give me another example:
Example 2: "Let's show that 4n^2 + 50n
- 10 = O(n^2). It is easy to see that: 4n^2 + 50n - 10 <= 4n^2 + 50n
for any n. Since 50n <= 50n^2 for n
= 50, 4n^2 + 50n - 10 <= 4n^2 + 50n^2 = 54n^2 for n >= 50. Thus, with c = 54 and N = 50, we have shown that 4n^2
+ 50n - 10 = O(n^2)."
This statement doesn't make sense: 50n <= 50n^2 for n >= 50.
Isn't any n going to make the 50n less than 50n^2? Not just greater than or equal to 50? Why did they even mention that 50n <= 50n^2? What does that have to do with the problem?
Also, 4n^2 + 50n - 10 <= 4n^2 + 50n^2 = 54n^2 for n >= 50 is going to be true no matter what n is.
And how in the world does picking numbers show that f(n) = O(g(n))?
Keep in mind that you're looking for "an upper bound on f(n) when n is sufficiently large." Thus, if you can show that f(n) is less than or equal to some cg(n) for values of n greater than N, this means cg(n) is an upper bound for f(n) and f(n)'s complexity is therefore O(g(n)).
The examples given are intended to show that the given function f(n) can never grow beyond c*g(n) for any n > N. By manipulating an initial upper bound so it can be expressed more simply (if 4n^2 + 50n is an upper bound on f(n) then so is 4n^2 + 50n^2, which is equal to 54n^2, which becomes your 54*g(n) where c = 54 and g(n) = n^2), the authors can show that f(n) is bounded by c*g(n), which has complexity O(g(n)) and therefore so does f(n).
The whole thing about picking numbers is just this: To make it easier. Because you're allowed to pick any numbers you like for N and c, the author just picks something, where it's most easy to see. And that's what you can also do (when writing an exam etc).
So while it would often be possible to use a smaller N, the reasoning would become a little bit harder (often requiring some previous knowledge about analysis - we've all learnt years before, that x doesn't grow as fast as x^2... But do you want to write down the analysis proof?)
Keep it simple, is the message :-) It's just a bit strange to get used to this at first.
50n <= 50n^2 for n >= 50
because if n is 50, then 50n is the same as n^2, because 50*50 equals 50^2.
Substituting n^2 for 50n we get
n^2 <= 50n^2 for n >= 50
which is obvious.
Probably the reason that they said 50n<=50n^2 for n>=50 is that if n is less than 1, than n^2 < n. Of course, if n is a positive integer, then yes 50n<=50n^2. In this case, it seems that n is assumed to be a positive integer, although the formal definition they give doesn't state that explicitly.
I can see why saying 50n<=50n^2 for n>=50 may seem a little silly. But it's still true. The book doesn't say 50n<=50n^2 holds ONLY for n>=50; that would be false.
As an analogy, if I say "all of my siblings speak English", that would be true, even though there are a lot of people who speak English who are not my siblings.
Regarding the proof, we might split it into different statements.
(1): 4n^2 + 50n - 10 <= 4n^2 + 50n (for all n)
(2): 4n^2 + 50n <= 4n^2 + 50n^2 (for all n>=50)
(3): 4n^2 + 50n^2 = 54 n^2 (for all n, including all n>=50)
(4): Therefore, 4n^2 + 50n - 10 <= 54n^2 for all n>=50
(5): Therefore, for f(n)=4n^2 + 50n - 10, g(n)=n^2, N=50, and c=54,
the statement f(n) <= c g(n) for all n >= N is true
(6): Therefore, by definition 4n^2 + 50n - 10=O(n^2).
It should be clear that each of these statements is true, either on its own (1,2,3), or as a result of the previous statements.
Formal definition:
f(n) = O(g(n)) means there exist c > 0 and n0 such that for any n >= n0 f(n) <= c*g(n)
f(n) = o(g(n)) means for any c > 0 there exist n0 such that for any n >= n0 f(n) <= c*g(n)
As you can note there are slightly different :)