See the following two functions.
A(n)
{ if(n<=1)
return;
else
return(A(n/4)+A(n/4)+A(n/4));
}
and the second one-
A(n)
{ if(n<=1)
return;
else
return(3*A(n/4));
}
Please, tell me the equation for both the functions with explanation and then bound it asymptotically.
Actually, why I am asking this question is because, I got an equation
T(n)=3T(n/4)+1
I used Masters and tree method (assuming first case)and got-
THETA(n^0.79)
But I wish to know why I can't assume this equation to be of 2nd case? One thing, I am sure about is that in both the cases, complexity varies as no. of recursive calls are different in both the case.
Please, help me understand it.
You're absolutely correct in your assertion that the recursion for the first algorithm's time, is
T(n) = 3 T(n / 4) + O(1).
It is also true that the first and second algorithm always return the same thing.
However, this is where the similarity ends. The second algorithm is composed more cleverly, by making a single call, and then multiplying. That is, while
return(A(n/4)+A(n/4)+A(n/4));
returns the same value as
return(3*A(n/4));
The latter makes only a single recursive call. Its recursion for the time, therefore is
T(n) = T(n / 4) + O(1)
(Here the last O(1) includes also the cost of multiplying the return value by 3, which doesn't matter w.r.t. the complexity.)
Related
Suppose we have a recursive function which only terminates if a randomly generated parameter meets some condition:
e.g:
{
define (some-recursive-function)
x = (random in range of 1 to 100000);
if (x == 10)
{
return "this function has terminated";
}
else
{
(some-recursive-function)
}
}
I understand that for infinite loops, there would not be an complexity defined. What about some function that definitely terminates, but after an unknown amount of time?
Finding the average time complexity for this would be fine. How would one go about finding the worse case time complexity, if one exists?
Thank you in advance!
EDIT: As several have pointed out, I've completely missed the fact that there is no input to this function. Suppose instead, we have:
{define (some-recursive-function n)
x = (random in range of 1 to n);
if (x == 10)
{
return "this function has terminated";
}
else
{
(some-recursive-function)
}
}
Would this change anything?
If there is no function of n which bounds the runtime of the function from above, then there just isn't an upper bound on the runtime. There could be an lower bound on the runtime, depending on the case. We can also speak about the expected runtime, and even put bounds on the expected runtime, but that is distinct from, on the one hand, bounds on the average case and, on the other hand, bounds on the runtime itself.
As it's currently written, there are no bounds at all when n is under 10: the function just doesn't terminate in any event. For n >= 10, there is still no upper bound on any of the cases - it can take arbitrarily long to finish - but the lower bound in any case is as low as linear (you must at least read the value of n, which consists of N = ceiling(log n) bits; your method of choosing a random number no greater than n may require additional time and/or space). The case behavior here is fairly uninteresting.
If we consider the expected runtime of the function in terms of the value (not length) of the input, we observe that there is a 1/n chance that any particular invocation picks the right random number (again, for n >= 10); we recognize that the number of times we need to try to get one is given by a geometric distribution and that the expectation is 1/(1/n) = n. So, the expected recursion depth is a linear function of the value of the input, n, and therefore an exponential function of the input size, N = log n. We recover an exact expression for the expectation; the upper and lower bounds are therefore both linear as well, and this covers all cases (best, worst, average, etc.) I say recursion depth since the runtime will also have an additional factor of N = log n, or more, owing to the observation in the preceding paragraph.
You need to know that there are "simple" formulas to calculate the complexity of a recursive algorithm, using of course recurrence.
In this case we obviously need to know what is that recursive algorithm, because in the best case, it is O(1) (temporal complexity), but in the worst case, we need to add O(n) (having into account that numbers may repeat) to the complexity of the algorithm itself.
I'll put this question/answer for more facility:
Determining complexity for recursive functions (Big O notation)
I would like some clarification regarding O(N) functions. I am using SICP.
Consider the factorial function in the book that generates a recursive process in pseudocode:
function factorial1(n) {
if (n == 1) {
return 1;
}
return n*factorial1(n-1);
}
I have no idea how to measure the number of steps. That is, I don't know how "step" is defined, so I used the statement from the book to define a step:
Thus, we can compute n ! by computing (n-1)! and multiplying the
result by n.
I thought that is what they mean by a step. For a concrete example, if we trace (factorial 5),
factorial(1) = 1 = 1 step (base case - constant time)
factorial(2) = 2*factorial(1) = 2 steps
factorial(3) = 3*factorial(2) = 3 steps
factorial(4) = 4*factorial(3) = 4 steps
factorial(5) = 5*factorial(4) = 5 steps
I think this is indeed linear (number of steps is proportional to n).
On the other hand, here is another factorial function I keep seeing which has slightly different base case.
function factorial2(n) {
if (n == 0) {
return 1;
}
return n*factorial2(n-1);
}
This is exactly the same as the first one, except another computation (step) is added:
factorial(0) = 1 = 1 step (base case - constant time)
factorial(1) = 1*factorial(0) = 2 steps
...
Now I believe this is still O(N), but am I correct if I say factorial2 is more like O(n+1) (where 1 is the base case) as opposed to factorial1 which is exactly O(N) (including the base case)?
One thing to note is that factorial1 is incorrect for n = 0, likely underflowing and ultimately causing a stack overflow in typical implementations. factorial2 is correct for n = 0.
Setting that aside, your intution is correct. factorial1 is O(n) and factorial2 is O(n + 1). However, since the effect of n dominates over constant factors (the + 1), it's typical to simplify it by saying it's O(n). The wikipedia article on Big O Notation describes this:
...the function g(x) appearing within the O(...) is typically chosen to be as simple as possible, omitting constant factors and lower order terms.
From another perspective though, it's more accurate to say that these functions execute in pseudo-polynomial time. This means that it is polynomial with respect to the numeric value of n, but exponential with respect to the number of bits required to represent the value of n. There is an excellent prior answer that describes the distinction.
What is pseudopolynomial time? How does it differ from polynomial time?
Your pseudocode is still pretty vague as to the exact details of its execution. A more explicit one could be
function factorial1(n) {
r1 = (n == 1); // one step
if r1: { return 1; } // second step ... will stop only if n==1
r2 = factorial1(n-1) // third step ... in addition to however much steps
// it takes to compute the factorial1(n-1)
r3 = n * r2; // fourth step
return r3;
}
Thus we see that computing factorial1(n) takes four more steps than computing factorial1(n-1), and computing factorial1(1) takes two steps:
T(1) = 2
T(n) = 4 + T(n-1)
This translates roughly to 4n operations overall, which is in O(n). One step more, or less, or any constant number of steps (i.e. independent of n), do not change anything.
I would argue that no you would not be correct in saying that.
If something is O(N) then it is by definition O(N+1) as well as O(2n+3) as well as O(6N + -e) or O(.67777N - e^67). We use the simplest form out of convenience for notation O(N) however we have to be aware that it would be true to say that the first function is also O(N+1) and likewise the second is as much O(n) as it wasO(n+1)`.
Ill prove it. If you spend some time with the definition of big-O it isn't too hard to prove that.
g(n)=O(f(n)), f(n) = O(k(n)) --implies-> g(n) = O(k(n))
(Dont believe me? Just google transitive property of big O notation). It is then easy to see the below implication follows from the above.
n = O(n+1), factorial1 = O(n) --implies--> factorial1 = O(n+1)
So there is absolutely no difference between saying a function is O(N) or O(N+1). You just said the same thing twice. It is an isometry, a congruency, a equivalency. Pick your fancy word for it. They are different names for the same thing.
If you look at the Θ function you can think of them as a bunch of mathematical sets full of functions where all function in that set have the same growth rate. Some common sets are:
Θ(1) # Constant
Θ(log(n)) # Logarithmic
Θ(n) # Linear
Θ(n^2) # Qudratic
Θ(n^3) # Cubic
Θ(2^n) # Exponential (Base 2)
Θ(n!) # Factorial
A function will fall into one and exactly one Θ set. If a function fell into 2 sets then by definitions all functions in both sets could be proven to fall into both sets and you really just have one set. At the end of the day Θ gives us a perfect segmentation of all possible functions into set of countably infinite unique sets.
A function being in a big-O set means that it exists in some Θ set which has a growth rate no larger than the big-O function.
And thats why I would say you were wrong, or at least misguided to say it is "more O(N+1)". O(N) is really just a way of notating "The set of all functions that have growth rate equal to or less than a linear growth". And so to say that:
a function is more O(N+1) and less `O(N)`
would be equivalent to saying
a function is more "a member of the set of all functions that have linear
growth rate or less growth rate" and less "a member of the set of all
functions that have linear or less growth rate"
Which is pretty absurd, and not a correct thing to say.
I got stuck with this two codes.
Code 1
int f(int n){
if (n <= 1){
return 1;
}
return f(n-1) + f(n-1);
}
Code 2 (Balanced binary search tree)
int sum(Node node){
if(node == null){
return 0;
}
return sum(node.left) + node.value + sum(node.right);
}
the author says the runtime of Code 1 is O(2^n) and space complexity is O(n)
And Code 2 is O(N)
I have no idea what's different between those two codes. it looks like both are the same binary trees
Well there's a mistake because the first snippet runs in O(2^n) not O(n^2).
The explanation is:
In every step we decrement n but create twice the number of calls, so for n we'll call twice with f(n-1) and for each one of the calls of n-1 we'll call twice with f(n-2) - which is 4 calls, and if we'll go another level down we'll call 8 times with f(n-3): so the number of calls is: 2^1, then 2^2, then 2^3, 2^4, ..., 2^n.
The second snippet is doing one pass on a binary tree and reaches every node exactly once, so it's O(n).
First of all, it's important to understand what N is in both cases.
In the first example it's pretty obvious, because you see it directly in the code. For your first case, if you build the tree of f(i) calls, you'll see that it contains O(2^N) elements. Indeed,
f(N) // 1 node
/ \
f(N-1) f(N-1) // 2 nodes
/ \ / \
f(N-2) f(N-2) f(N-2) f(N-2) // 2^2 nodes
...
f(1) ........ f(1) // 2^(N-1) nodes
In the second case, N is (most likely) a number of elements in the tree. As you may see from the code, we walk through every element exactly once - you may realize it as you see that node.value is invoked once for each tree node. Hence O(N).
Note that in such tasks N normally means the size of the input, while what the input is depends on your problem. It can be just a number (like in your first problem), a one-dimensional array, a binary tree (like in your second problem), or even a matrix (although in the latter case you may expect to see explicit statement like "a matrix with a size M*N").
So your confusion probably comes from the fact that the "definition of N" differs between those two problems. In other words, I might say that n2 = 2^n1.
The first code is indeed O(2^n).
But the second code cannot be O(n), because there is no n there. That's a thing which many forget and usually they assume what n is without clarifying it.
In fact you can estimate growth speed of anything based on anything. Sometimes it's a size of input (which in the first code is O(1) or O(log n) depending on usage of big numbers), sometimes just on argument if it's numeric.
So when we start thinking about what time and memory depend on in the second code we can get these things:
time=O(number_of_nodes_in_tree)
time=O(2^height_of_tree)
additional_space=O(height_of_tree)
additional_space=O(log(number_of_nodes)) (if the tree is balanced)
All of them are correct at the same time - they just relate something to different things.
You’re confused between the “N” of the two cases. In the first case, the N refers to the input given. So for instance, if N=4, then the number of the functions being called is 2^4=16. You can draw the recursive map to illustrate. Hence O(2^N).
In the second case, the N refers to the number of nodes in the binary tree. So this N has no relation with the input but the amount of nodes that already exists in the binary tree. So when user calls the function, it visits every node exactly once. Hence O(N).
Code 1:
The if() statement runs n times according to whatever is passed into the parameter, but the the function call itself n-1 times. To simplify:
n * (n-1) = n^2 - n = O(n^2 - n) = O(n^2)
Code 2:
The search traverses every element of the tree only once, and the function itself doesn't have any for(). Since there are n items and they are visited only once, it is O(n).
For Code 2, to determine the Big O of a function, didn't we have to consider the cost of the recurrence and also how many times the recurrence was run?
If we use two approach to estimate the Big O using recursive tree and master theorem:
Recursive tree:
total cost in each level will be cn for each level as the number of recursive call and the fraction of input are equal, and the level of tree is lg(n) since it's a balanced binary search tree. So the run time should be nlg(n)?
Master Theorem:
This should be a case 2 since f(n) = n^logbase a (b). So according to the master theorem, it should be nlg(n) running time?
We can think of it as O(2^Depth).
In the first example: The depth is N, which happens to be the input of the problem mentioned in the book.
In the second example: It is a balanced binary search tree, hence, it has Log(N) levels (depth). Note: N is the number of elements in the tree.
=> Let's apply our O(2^Depth).. O(2^(Log(N)) = O(N) leaving us with O(N) complexity.
Reminder:
In computer science we usually refer to Log2(n) as Log(n).
The logarithm of x in base b is the exponent you put on b to get x as a result.
In the above complexity: O(2^(Log(N), we're raising the base 2 to Log2(N) which gives us N. (Check the two reminders)
This link can be useful.
In my algorithm and data structures class we were given a few recurrence relations either to solve or that we can see the complexity of an algorithm.
At first, I thought that the mere purpose of these relations is to jot down the complexity of a recursive divide-and-conquer algorithm. Then I came across a question in the MIT assignments, where one is asked to provide a recurrence relation for an iterative algorithm.
How would I actually come up with a recurrence relation myself, given some code? What are the necessary steps?
Is it actually correct that I can jot down any case i.e. worst, best, average case with such a relation?
Could possibly someone give a simple example on how a piece of code is turned into a recurrence relation?
Cheers,
Andrew
Okay, so in algorithm analysis, a recurrence relation is a function relating the amount of work needed to solve a problem of size n to that needed to solve smaller problems (this is closely related to its meaning in math).
For example, consider a Fibonacci function below:
Fib(a)
{
if(a==1 || a==0)
return 1;
return Fib(a-1) + Fib(a-2);
}
This does three operations (comparison, comparison, addition), and also calls itself recursively. So the recurrence relation is T(n) = 3 + T(n-1) + T(n-2). To solve this, you would use the iterative method: start expanding the terms until you find the pattern. For this example, you would expand T(n-1) to get T(n) = 6 + 2*T(n-2) + T(n-3). Then expand T(n-2) to get T(n) = 12 + 3*T(n-3) + 2*T(n-4). One more time, expand T(n-3) to get T(n) = 21 + 5*T(n-4) + 3*T(n-5). Notice that the coefficient of the first T term is following the Fibonacci numbers, and the constant term is the sum of them times three: looking it up, that is 3*(Fib(n+2)-1). More importantly, we notice that the sequence increases exponentially; that is, the complexity of the algorithm is O(2n).
Then consider this function for merge sort:
Merge(ary)
{
ary_start = Merge(ary[0:n/2]);
ary_end = Merge(ary[n/2:n]);
return MergeArrays(ary_start, ary_end);
}
This function calls itself on half the input twice, then merges the two halves (using O(n) work). That is, T(n) = T(n/2) + T(n/2) + O(n). To solve recurrence relations of this type, you should use the Master Theorem. By this theorem, this expands to T(n) = O(n log n).
Finally, consider this function to calculate Fibonacci:
Fib2(n)
{
two = one = 1;
for(i from 2 to n)
{
temp = two + one;
one = two;
two = temp;
}
return two;
}
This function calls itself no times, and it iterates O(n) times. Therefore, its recurrence relation is T(n) = O(n). This is the case you asked about. It is a special case of recurrence relations with no recurrence; therefore, it is very easy to solve.
To find the running time of an algorithm we need to firstly able to write an expression for the algorithm and that expression tells the running time for each step. So you need to walk through each of the steps of an algorithm to find the expression.
For example, suppose we defined a predicate, isSorted, which would take as input an array a and the size, n, of the array and would return true if and only if the array was sorted in increasing order.
bool isSorted(int *a, int n) {
if (n == 1)
return true; // a 1-element array is always sorted
for (int i = 0; i < n-1; i++) {
if (a[i] > a[i+1]) // found two adjacent elements out of order
return false;
}
return true; // everything's in sorted order
}
Clearly, the size of the input here will simply be n, the size of the array. How many steps will be performed in the worst case, for input n?
The first if statement counts as 1 step
The for loop will execute n−1 times in the worst case (assuming the internal test doesn't kick us out), for a total time of n−1 for the loop test and the increment of the index.
Inside the loop, there's another if statement which will be executed once per iteration for a total of n−1 time, at worst.
The last return will be executed once.
So, in the worst case, we'll have done 1+(n−1)+(n−1)+1
computations, for a total run time T(n)≤1+(n−1)+(n−1)+1=2n and so we have the timing function T(n)=O(n).
So in brief what we have done is-->>
1.For a parameter 'n' which gives the size of the input we assume that each simple statements that are executed once will take constant time,for simplicity assume one
2.The iterative statements like loops and inside body will take variable time depending upon the input.
Which has solution T(n)=O(n), just as with the non-recursive version, as it happens.
3.So your task is to go step by step and write down the function in terms of n to calulate the time complexity
For recursive algorithms, you do the same thing, only this time you add the time taken by each recursive call, expressed as a function of the time it takes on its input.
For example, let's rewrite, isSorted as a recursive algorithm:
bool isSorted(int *a, int n) {
if (n == 1)
return true;
if (a[n-2] > a[n-1]) // are the last two elements out of order?
return false;
else
return isSorted(a, n-1); // is the initial part of the array sorted?
}
In this case we still walk through the algorithm, counting: 1 step for the first if plus 1 step for the second if, plus the time isSorted will take on an input of size n−1, which will be T(n−1), giving a recurrence relation
T(n)≤1+1+T(n−1)=T(n−1)+O(1)
Which has solution T(n)=O(n), just as with the non-recursive version, as it happens.
Simple Enough!! Practice More to write the recurrence relation of various algorithms keeping in mind how much time each step will be executed in algorithm
I have an algorithm called rec(n):
rec(n)
if (n=0) return 1
else
i=rec(n-1)
A[n]=i
return i
I was looking at it, and from what I can see it seems like no matter what you put in there it'll always return a value of 0, so I assumed that the recurrence relation would be a(n)=a(n-1) and the time complexity would be constant (i.e. O(1)), but I'm hesitant about my interpretation of it. Could anyone help me out?
You are right that no matter what the value of n is, rec(n) will always return 1. This can be proved trivially by induction using the relation rec(n) = rec(n-1) with base case rec(0) = 1.
On the other hand, the complexity of your function rec(n) should be O(n). This is because when you compute the value of rec(n), you first need to compute the value of rec(n-1); but in order to find rec(n-1) you need to compute the value of rec(n-2) and so on.
Therefore when computing the value of rec(n), you need to invoke rec() n times and hence the complexity is O(n).