I came across this passage on page 47 of Introduction to Algorithms by Cormen et al.:
The number of anonymous functions in an expression is understood to be equal to the number of times the asymptotic notation appearrs. For example in the expression:
Σ (i=1 to n) O(i)
there is only a single anonymous function (a function of i). This expression is not the same as O(1) + O(2) + ... + O(n), which doesn't really have a clean interpretation.
What does this mean?
I think they're saying that when they use that notation (sum of a big-O), it means there's a single O(i) function (call it f(i)), and then the expression refers to the sum from 1 to n of that function.
This isn't the same thing as if there were n different functions (call them f_1(i) to f_n(i)), each of which is O(i), and then the expression refers to the sum of f_1(1) + f_2(2) + ... + f_n(n). That latter thing is not what the notation means.
i think you should understand "The number of anonymous functions in an expression is understood to be equal to the number of times the asymptotic notation appears" at first.
it means if
Σ (i=1 to n) O(i)
so anonymous functions is one equals to O(i)
such as Σ (i=1 to n) f1(i)
and if Σ (i=1 to n) O(i)+O(i)
anonymous functions should have two functions equals to O(i)+O(i)
such as Σ (i=1 to n) f1(i)+f2(i)
The following is according to my understanding:
The passage simply states that.
This is what everyone first understands and it is the correct understanding of that cryptic passage, but the problem is WHY?, since we all are used to expand the sigma notations in the usual (and correct) way as in the expression above.
It turns out there is an exception when it comes to asymptotic notations like , etc. Recall that these asymptotic notations are all rough approximations for some anonymous function; due to this fact (the passage intends to convey that), before you expand and compute those sigma notations containing some asymptotic notation, you must firstly
figure out/evaluate/find out exactly that anonymous function and
substitute it for the asymptotic notation in the sigma expression and
only then expand and compute the sigma notation.
Thus:
But now let’s suppose, for the sake of argument, that the expression
is valid. But immediately we will face a problem here: since each of O(1), O(2), O(3),…, O(n) has a different value within its parentheses (1, 2, etc) and each subsequent number within parentheses is bigger than the previous one, it appears that each of these Big-Os represent a different anonymous function and each of these different anonymous functions grow faster than the previous different function.
Related
I'm a teacher and told my students that the Big O of an expression such as y=15x+8 is O(1), however when we learned prefix and postfix, we discussed that evaluating these expressions is O(N) because you have to go through each character of the equation (assuming you're given a string).
One student asked how we said that evaluating an expression is O(1) if behind the scenes there must be either an infix, postfix or prefix evaluation being done.
I'm not sure what to answer.
An expression does not have any time complexity as such. An algorithm to solve a problem can have a time complexity. So it all depends on what you define as your problem and what you define to be relevant parameters of complexity. If you have a fixed assignment such as
y := 15 * x + 8;
the problem can be defined as "compute the value of 15 * x + 8, with input parameter x". So here you want to express time complexity as a function dependent on x. The time complexity is O(1), assuming we are talking about standard 32/64-bit arithmetic computations and otherwise O(log x), if this is arbitrary-precision arithmetic.
However, if you regard the size of the expression as variable, the problem becomes "compute the value of an arithmetic expression tree with k nodes, where k is an input parameter". This is a different problem, and has a different complexity, as you correctly pointed out.
i am stuck on this one and i don't know how to solve it, no matter what i try i just can't find a way to play with the function so i can represent it in a way that will allow me to find a g(n), so that g(n) is T(n)∈Θ(g(n))
the function i am having trouble with is:
$T(n)=4n^4T(\sqrt n) +(n^6lgn+3lg^7n)(2n^2lgn+lg^3n)$
additionally, if you can - could you please check if i am on the right path with:
$T(n)=T(n-1)+\frac{1}{n}+\frac{1}{n^2}$
to solve it i tried to use: $T(n)-T(n-1)=\frac{1}{n}+\frac{1}{n^2}$ iff $(T(n)-T(n-1))+(T(n-1)-T(n-2))+\ldots+(T(2)-T(1))=\frac{1}{n}+\frac{1}{n-1}+...+\frac{1}{n^2}+\frac{1}{\left(n-1\right)^2}+....$ iff $(T(n)-T(n-1))+(T(n-1)-T(n-2))+\ldots+(T(2)-T(1))=T(n)=T(1)+\sum_{k=2}^n\frac{1}{n}+\sum_{k=2}^n\frac{1}{n^2}$ and then using the harmonic series formula. however i don't know how to continue and finish it and find the asymptotic boundaries to solve it
i hope that on the second i am on the right path. however i don't know how to solve the first one at all. if i've done any mistakes, please show me the right way so i can improve my mistakes.
thank you very much for your help
sorry that for some reason math doesn't show correctly here
Following on from the comments:
Solving (2) first since it is more straightforward.
Your expansion attempt is correct. Writing it slightly differently:
A, harmonic series - asymptotically equal to the natural logarithm:
γ = 0.57721... is the Euler-Mascheroni constant.
B, sum of inverse squares - the infinite sum is the famous Basel problem:
Which is 1.6449.... Therefore since B is monotonically increasing, it will always be O(1).
The total complexity of (2) is simply Θ(log n).
(1) is a little more tedious.
Little-o notation: strictly lower complexity class, i.e.:
Assume a set of N functions {F_i} is ordered in decreasing order of complexity, i.e. F2 = o(F1) etc. Take a linear combination of them:
Thus a sum of different functions is asymptotically equal to the one with the highest growth rate.
To sort the terms in the expansion of the two parentheses, note that
Provable by applying L'Hopital's rule. So the only asymptotically significant term is n^6 log n * 2n^2 log n = 2n^8 log^2 n.
Expand the summation as before, note that i) the factor 4n^4 accumulates, ii) the parameter for the m-th expansion is n^(1/(2^m)) (repeated square-root).
The new term added by the m-th expansion is therefore (will assume you know how to do this since you were able to do the same for (2)):
Rather surprisingly, each added term is precisely equal to the first.
Assume that the stopping condition for recursive expansion is n < 2 (which of course rounds down to T(1)):
Since each added term t_m is always the same, simply multiply by the maximum number of expansions:
Function (1) is
A colleague of mine has asked me a question: Is the set o(1) (little o notation) empty?
My question is: Is o(1) an empty set? If not, is there a program that has o(1) time complexity?
Reminder, definition of little-o by Cormen:
A function f(n) is said to be in o(g(n)) if for any positive
constant c>0, there exists a constant n0 > 0 such that 0 <=f(n) < cg(n) , for all n>= n0.
Intuitively, if f(n) is in o(g(n)) if it is in O(g(n)), but this bound is NOT tight.
The set o(1) is not empty.
It is first important to remember that f(x) is in o(g(x)) if and only if
lim_x->infinity { f(x) / g(x)} = 0
For non zero g(x)
But, more important, what is the set of candidate f(x)?
Some define it over all real functions [1], i.e. f:R->RU{U} (Where U is undefined for some values of the functions). This means, we can use any real to real function, including the function f(x)=1/x.
We can also see that g(x)=1 is a non zero function, and indeed:
lim_x->infinity { 1/x / 1} = 0
This means, o(1) includes the function f(x)=1/x, and we can conclude the set is none empty.
Knuth also refers the function g(n) = n^-1 as a valid function, and uses O(n^-1) in his explanation of big O,Omega and Theta (1976)
Others, Cormen is one of them, define the set as f:N->N, where N={0,1,...}, and this also includes f(x)=0, which again holds the condition for being o(1).[2]
There is no algorithm with complexity function T(n) in o(1)
While little o notation is defined over reals, our complexity functions for algorithms are not. They are defined over natural numbers [3]. You either do the instruction, or don't. You cannot do half of an instruction, or e-1 instruction. This means, the set of complexity functions are f:N->N. Since there is no such thing as "empty program" that has no instructions (recall that the overhead of calling it itself takes time), it even narrows this range to f:N->N\{0}.
In other words, for any complexity function of an algorithm T(n), and for all n>0, T(n)>0.
We can now go back to our formula:
lim_x->infinity { T(x) / 1} >= lim_x->infinity { 1 / 1} = 1 > 0
This shows us there is no positive natural function in o(1), and we can conclude no algorithm has a complexity function that is in o(1).
Foot Notes:
(1) If you are unsure about it, recall in Taylor series, at some point we stop adding the infinite series, and just mention it is O(x^n). The function we "hide" in this big O notation is not over the natural numbers.
(2)If we however define the set N+={1,2,...} to be the set of positive natural numbers, and o(g(n)) to be a subset of positive natural functions, o(1) is an empty set, with a proof identical to the one showing no algorithm has this complexity.
(3) Well, for average cases, the image can be a non natural number, but we'll assume worst case complexity here, though the claim can still hold for average case, since there is no empty program.
The function f(n)=0 is in o(1) and so o(1) is not empty. Because for every c>0, f(n) < c * 1.
It's a matter of opinion (or definition) whether a program's time complexity can be o(1). If you think that a program can exist with no basic operations then it would have time complexity in o(1). If you think a program can't exist with no basic operations, then it will always take at least 1 time unit no matter the input, and picking c=0.5 in the definition of little-o gives you a proof that its time complexity is not o(1).
From the definition of little o follows that for this condition to meet (being o(1)),there must be algorithm that completes in arbitrary short time.
This contradicts with definition of Turing machine that requires "infinite tape marked out into squares (of finite size)". Only solution for this would be empty Turing program executing 0 instruction.
but, such program can't be build, because it would require machine, that starts in terminated state and thus can not execute any other program and is not Turing machine.
are there a limited amount of basic O Notations, considering you are meant to 'distil' them down to their most important part?
O(n^2):
O(n):
O(1):
O(log n) logarithmic
O(n!) factorial
O(na) polynomial
Or are you expected to work out variations such as O(n^4) etc... and if so, is that the only exception? the power of X one?
Generally, you distill Big-O notation (and related Bachman-Landau notations like Big-Theta and Big-Omega) down to the operation of the fastest-growing N term. So, you remove/simplify lesser terms (N2 + N == O(N2)) and nonvariable coefficients of the term (O(4N2) == O(N2)), but NOT powers or exponent bases (O(34N) == O(3N)). You also don't strip variable coefficients; NlogN is NlogN, NOT logN or N.
So, you will normally only see numbers in a Big-Oh notation if if the complexity is polynomial (power of N) or exponential (Nth power of a base). The most common Big-Oh notations are much as you show, with the addition of NlogN (VERY common).
However, if you are differentiating between two algorithms of equal general complexity, you MAY add lesser terms and/or coefficients back in to demonstrate the relative difference; an algorithm that performs linearly but has double the instructions of another might be described as O(2N) when comparing it with the other O(N) algorithm. However, taken individually, both algorithms are linear (O(N)).
Some Big-O notations are not algebraic, and may involve multiple variables in therir simplest general-case form. The counting sort, for instance, is complexity O(Max(N,M)), where N is the number of elements in the list, and M is the range of those elements. Often it is possible to reduce this in specific cases by defining M in terms of N and thus reducing to a single variable (if the list in question is of the first N squares, M = N2-1), but in the general case both variables are independent and significant. BucketSort's complexity is officially O(N), but really it's more like O(NlogM) where M is the maximum value of the list of N elements. M is usually considered insignificant, but that depends on the values you normally sort (sorting 5 values each in the billions will require more loops to compare each power of 10 than traversals through the list to put them in the buckets) and on the radix used (RadixSort is a base-2 BucketSort; again, sorting values with a greater log2 value will require more loops than traversals).
The Big-O notation is a way to provide an upper bound on the limiting behaviour of a function. There are no restrictions on its functional form. However, there are certain conventions, as explained by Wikipedia:
In typical usage, the formal definition of O notation is not used directly; rather, the O notation for a function f(x) is derived by the following simplification rules:
If f(x) is a sum of several terms, the one with the largest growth rate is kept, and all others omitted.
If f(x) is a product of several factors, any constants (terms in the product that do not depend on x) are omitted.
There are, of course, some functional forms that show up more frequently that others. Some common classes are listed here.
No, the number of different O-classes is not finite.
As you already mentioned O(n^x) describes a different set for every x. And that is not the only "exception". O(x^n) is also a different set for every x. Likewise O(n^n), O(n^n^n), O(n^n^n^n) etc. are all different sets (and you can of course continue that ad infinitum).
In general, you split the expression into a sum of products, keep the largest term, and divide by constants to simplify it as much as possible.
ex:
n(2n+3log(n)) => 2n^2+3nlog(n) => 2n^2 => n^2
(n+1)(2nlog(n)+n) => 2n^2log(n)+n^2+2nlog(n)+n => 2n^2log(n) => n^2log(n)
Order notation question, big-o notation and the like:
What does the max and min of a function mean in terms of order notation?
for example:
DEFINITION:
"Maximum" rules: Suppose that f(n) and g(n) are positive functions for all n > n0.
Then:
O[f(n) + g(n)] = O[max (f(n),g(n)) ]
etc...
I need to use these definitions to prove something for homework.. thanks for the help!
EDIT: f(n) and g(n) are supposed to represent running times of algorithms with respect to input size
With Big-O notation, you're talking about upper bounds on computation. This means that you are only interested in the largest term of the combined function as n (the variable) tends to infinity. What's more, you drop any constant multipliers as well since the formal definition of the notation allows you to discard those parts, which is important as it allows you to focus on the algorithm's behavior and not on the implementation of the algorithm.
So, we are combining two functions through summation. Well, there are two cases (strictly three, but it's symmetric):
One function is of higher order than the other. At that point, the higher order function dominates the lesser; you can pretend that the lesser doesn't exist.
Both functions are of the same order. This is just like you are doing some kind of ratio-ed sum of the two (since we've already thrown away the scaling factors) but then you just end up with the same factor and have just changed the scaling factors a bit.
The net result looks very much like the max() function, even though it's really not (it's actually a generalization of max over the space of functions), so it's very convenient to use the notation.
It is a regular max between natural numbers. f is a function mapped to numbers [f:N->N], and so is g.
Thus, f(n) is in N, and so max(f(n),g(n)) is just standard max: f(n) > g(n) ? f(n) : g(n)
O[max (f(n),g(n)) ] means: which ever is more 'expensive': f or g: it is the upper bound.