I have an implementation of an algorithm that runs in O(n log n), for n=10^7 the algorithm takes 570 ms. Does anyone know how to find the constant part (C) of my algorithms running time? I would like to have this so I can calculate how long the algorithm 'should' take for an arbitrary input size.
I don't think you can calculate it exactly, but if you know for sure that the complexity is O(n log n) then I would recommend a simple proportion as an estimate your run time:
10^10 log 10^10 unknown run time
--------------- = ----------------
10^7 log 10^7 570 ms
In this case, that should be about 1428.6 * 570 ms =~ 814 sec.
It's not exactly mathematically correct, but if you don't have multiple data points to try to fit to a curve to figure out the various constants, it's not an unreasonable place to start.
If you know that the asymptotic complexity of an algorithm is O(n log n), then with just a single data point you can't (exactly) determine the runtime on future operations. Imagine, for example, that you have an algorithm that you know runs in time O(n) and on an input of size N, the runtime is T. You can't exactly predict what the runtime is going to be on an input of size 2T, because it's unclear how much of T is explained by the slope of the linear function and how much is explained by the intercept.
If you assume that N is "large enough" that most of the runtime T comes from the slope, then you can make a reasonable estimate of the runtime of the algorithm on a future input. Specifically, since the function grows linearly, you can assume that if you multiply the size of the input by some constant k, then the runtime ought to be Tk. In your case, the function n log n grows mostly linearly. Since log grows very slowly, for large enough n its growth is extremely flat. Consequently, if you think that N is "large enough," you can estimate the runtime on an input of size kN by just scaling the runtime on size N by a factor of k.
To be much more accurate, you could also try gathering more data points about the runtime and doing a regression. In the linear case, if you know two accurate data points, you can recover the actual linear function and then extrapolate to get very accurate runtime predictions. With something of the form n log n, it's probably good to assume the runtime has the form c0 n log n + c1 n + c2 n. If you gather enough data points, you could probably plug this into Excel and recover the coefficients, from which you could extrapolate very accurately.
Hope this helps!
Related
What's the easiest way to get the time-complexity of a function when the input and time per input is given?
For example:
n | 10 20 40 80 160 320 640
time | 1 3 9 60 475 3732 29835
How could I find the time-complexity of this table?
Assuming, you are interested in polynomial complexities (O(n), O(n1.5), O(n2) ...).
The easiest way to estimate the time-complexity of your function implementation would be to look at the last two largest datapoints.
You doubled (*2) the input size n from 320 to 640.
And your execution time increased by a factor of 8 (29835/3732 =7.994...).
Thus, your time complexity estimate is O(n3) since 23=8.
One should pay attention, that this estimate is correct only if you tested it on large enough datasets (n is large) and your lower-order terms do not influence a solution significantly anymore. Moreover, this is a time-complexity of your implementation of the algorithm, not the algorithm itself.
Another useful technique to understand the complexities of your code would be plotting them (t versus n) on a log-log plot. Since the degree of the polynomial determining the complexity will become just the slope of the line on log-log plot, that can give you an understanding of what the complexity is and if you reached the asymptotic region or not. In addition, such plots can give you an idea of your lower order terms, log-terms and etc.
However, usually, you know what is the supposed complexity of the algorithm by some theoretical analysis, and then you may compare your actual timings to the theoretical prediction. That allows to avoid a lot of pitfalls during the estimation process.
I know that O(log n) refers to an iterative reduction by a fixed ratio of the problem set N (in big O notation), but how do i actually calculate it to see how many iterations an algorithm with a log N complexity would have to preform on the problem set N before it is done (has one element left)?
You can't. You don't calculate the exact number of iterations with BigO.
You can "derive" BigO when you have exact formula for number of iterations.
BigO just gives information how the number iterations grows with growing N, and only for "big" N.
Nothing more, nothing less. With this you can draw conclusions how much more operations/time will the algorithm take if you have some sample runs.
Expressed in the words of Tim Roughgarden at his courses on algorithms:
The big-Oh notation tries to provide a sweet spot for high level algorithm reasoning
That means it is intended to describe the relation between the algorithm time execution and the size of its input avoiding dependencies on the system architecture, programming language or chosen compiler.
Imagine that big-Oh notation could provide the exact execution time, that would mean that for any algorithm, for which you know its big-Oh time complexity function, you could predict how would it behave on any machine whatsoever.
On the other hand, it is centered on asymptotic behaviour. That is, its description is more accurate for big n values (that is why lower order terms of your algorithm time function are ignored in big-Oh notation). It can reasoned that low n values do not demand you to push foward trying to improve your algorithm performance.
Big O notation only shows an order of magnitude - not the actual number of operations that algorithm would perform. If you need to calculate exact number of loop iterations or elementary operations, you have to do it by hand. However in most practical purposes exact number is irrelevant - O(log n) tells you that num. of operations will raise logarythmically with a raise of n
From big O notation you can't tell precisely how many iteration will the algorithm do, it's just estimation. That means with small numbers the different between the log(n) and actual number of iterations could be differentiate significantly but the closer you get to infinity the different less significant.
If you make some assumptions, you can estimate the time up to a constant factor. The big assumption is that the limiting behavior as the size tends to infinity is the same as the actual behavior for the problem sizes you care about.
Under that assumption, the upper bound on the time for a size N problem is C*log(N) for some constant C. The constant will change depending on the base you use for calculating the logarithm. The base does not matter as long as you are consistent about it. If you have the measured time for one size, you can estimate C and use that to guesstimate the time for a different size.
For example, suppose a size 100 problem takes 20 seconds. Using common logarithms, C is 10. (The common log of 100 is 2). That suggests a size 1000 problem might take about 30 seconds, because the common log of 1000 is 3.
However, this is very rough. The approach is most useful for estimating whether an algorithm might be usable for a large problem. In that sort of situation, you also have to pay attention to memory size. Generally, setting up a problem will be at least linear in size, so its cost will grow faster than an O(log N) operation.
I am undertaking the algorithms course on Coursera, there is a section where the author mentions the following
the running time of weighted quick union with path compression is
going be linear in the real world and actually could be improved to
even a more interesting function called the Ackermann function, which
is even more slowly growing than lg. And another point about this
is it seems that this is so close to being linear that is time proportional to N instead of time proportional to N times the
slowly growing function in N. Is there a simple algorithm that is
linear? And people, looked for a long time for that, and actually it
works out to be the case that we can prove that there is no such
algorithm. (emphasis added)
(You can find the entire transcript here)
In all other sources including Wikipedia "linear" is used when time increases proportionally with the input size, and in weighted quick-union with path compression this is certainly not the case.
What exactly is meant by "linear in the real world" here?
The runtime of m operations on a union-find data structure with path compression and union-by-rank is O(mα(m)), where α(m) is the inverse Ackermann function. This function is so slowly-growing that you cannot express an input to it for which the output is 6 in scientific notation. In other words, for any possible value of m that fits into the universe (or even that has size around 2num atoms in the universe), we have that α(m) ≤ 5. Therefore, for any "reasonable" input the cost of m operations will be O(m · 6) = O(m), which is linear.
Of course, the runtime isn't linear because α(m) does indeed grow, just very, very slowly. However, it's usually fine to approximate the runtime as O(m) because there's no possible way you'd ever notice the runtime of the function deviating from a simple linear function.
Hope this helps!
Here are some chunks from the transcript:
And what was proved
by Hopcroft Ulman and Tarjan was that if
you have N objects, any sequence of M
union and find operations will touch the
array at most a c (N + M lg star N) times.
And now, lg N is kind of a funny function....
And another point
about this is it< /i> seems that this is
so close to being linear that is t ime
proportional to N instead of time
proportional to N times the slowly growing
function in N.
(end quote)
You are pointing out that the cost of an individual operation grows very slowly with the number of objects, but they are looking at how the total cost of a number of operations grows with the number of objects involved so N times a per-operation cost that grows only very slowly with N is still just over linear in N.
I have a homework problem:
If an algorithm takes 0.5 ms for an input size of 100, how long will it take for inputs of 500, 1,000 and 10,000 if it is:
Linear,
O(N log N),
Quadratic,
Cubic,
Exponential.
I understand the basic concepts here, I'm just unsure of how to approach the question mathematically. I'm tempted to simply calculate the amount of time it takes to do process each individual item of input (for instance, in a) I'll divide 0.5 by 100 to get .05, which I'll then multiply by 500, 1000 and 1000 to find how long it will take to process inputs of those sizes.
While this is simple for the linear calculations, and also pretty straightforward for quadratic and cubic ones, I don't quite understand how to apply this strategy to (N log N) or exponential functions: What value do I use as the base of the exponent?
For (N log N), is it as simple as calculating c = 100 * log(100) = 200; then calculating c = 500 * log(500) = 1349, which is 6.745 times 200, then multiplying 6.745 by .5 to arrive at 3.3725 as a final answer?
This is a bad exercise:
It teaches you to extrapolate from a single datapoint.
It teaches you to apply the results of asymptotic analysis where they can't be applied.
To expand on the latter, you cannot predict the performance of a system when given specific inputs based on an asymptotic bound because of hidden terms. Many algorithms have hidden constant and linear terms that dominate the run time for small input sizes like n=100.
But if you are supposed to ignore these facts, your approach is correct.
Suppose the function's runtime is given by T(n). You can solve these problems by doing the following:
Write a general expression for T(n) in terms of the function's growth rate, given there is some hidden constant multiplier. For example, for the n log n case, write T(n) = c n log n for some constant c.
Solve for c using the one data point you have; T(100) is given to you.
Plug in values of n as needed to estimate the runtime.
This works for any growth rate.
A quick note: "exponential" is not precise enough for you to make a meaningful estimate. 2^n and 100^n are both exponential but the latter grows significantly faster than the former. In computer science, typically logs use base 2, though there's no general conventions for exponents.
Hope this helps!
I am reading algorithm analysis topic. Here is the text snippet from the book
When n doubles, the running time goes up by a factor of 2 for linear
programs, 4 for quadratic programs, and 8 for cubic programs.
Programs that run in logarithmic time take only an additive constant
longer when n doubles, and programs that run in O(n log n) take
slightly more than twice as long to run under the same circumstances.
These increases can be hard to spot if the lower-order terms have
relatively large coefficients and n is not large enough.
My question is what does author mean lower-order terms have relatively large coefficients? Can any one explain with example
Thanks!
Suppose your algorithm actually executes n^2 + 1000 n computations when run on n elements. Now for n = 1 you need 1001 computations, and for n = 2 you need 2004. The difference from linear growth is tiny, and you can hardly spot the quadratic contribution!
Asymptotically, however, your algorithm takes O(n^2) steps, so asymptotically (as n gets large) doubling the input size quadruples your runtime. But for our small value, doubling from 1 to 2 did not quadruple the runtime! The lower-order term is n, and its coefficient (1000) is large compared to the coefficient of the leading-order termn^2 (which is 1).
This shows how the asymptotic complexity does not say anything about particular, especially small values. It's merely a limiting statement about the behaviour as n gets large.
When using O notation, you specify the largest term of the function that is your performance bound. For example, if the performance was always bound by f = c3n3 + c2n2 + c1n + c0, you would say that is is O(n3). The author is saying that when n is small, the coefficients may have a larger impact than n on the performance, for example if c2 were very large and c3 very small, the performance may appear to be O(n2) until the size of n dominates the coefficients if you only go by the relative performance for specific small instances of n.
Asymptotic notation refers to the bounds of the runtime as n->infinity. So, a function that is O(n log n) may have an actual runtime of .1*n log n + 100000*n.
In this case, the 100000*n term is the "lower-order term". As n->infinity, this term is overpowered by the .1*n log n term.
However, as you can see, for small n, the 100000*n term will dominate the runtime.
For instance if you have an O(n) algorithm at lower scales you could have T(n) = 490239n + (insert ridiculous constant here) which means that the performance would look bad but as the scales increase you see that the increase is always linear.
Real world example is merge sort, O(n logn) problem is that recursion has a computational cost or overhead which is a factor of n which is a smaller order than nlogn so it gets discarded in the Big-O, problem is that that factor gets to be quite large as well and affects performance.