Suppose I have 3 algoithrms A,B and C to process n records.
algorithm A takes 80n + 40 steps
algorithm B takes n^2 + 30n steps
algorithm C takes 2^n steps
Decide which algorithms is most efficient when performing
i) 10 < n < 50
The way I would solve this problem is by assuming n is equals to a value for example
for i) Assume that n = 20
so
algo A - 80(20) + 40 = 1640 steps
algo B - 20^2 = 400 steps
algo C - 2^20 = 1048576 steps
therefore algo B is most efficent.
I am not really sure whether I have evaluated the 3 algorithms performance correctly because I am just substituting a n with a value instead of using Big O notation?
Please advise. thanks
Big-O notation deals with n that is arbitrary large, i.e. in order to evaluate O(n) the expression should be calculated for n-->infinity. In your case n is given, thus the overall running time can be precisely calculated, exactly the way you did it.
Related
I try to understand a formula when we should use quicksort. For instance, we have an array with N = 1_000_000 elements. If we will search only once, we should use a simple linear search, but if we'll do it 10 times we should use sort array O(n log n). How can I detect threshold when and for which size of input array should I use sorting and after that use binary search?
You want to solve inequality that rougly might be described as
t * n > C * n * log(n) + t * log(n)
where t is number of checks and C is some constant for sort implementation (should be determined experimentally). When you evaluate this constant, you can solve inequality numerically (with uncertainty, of course)
Like you already pointed out, it depends on the number of searches you want to do. A good threshold can come out of the following statement:
n*log[b](n) + x*log[2](n) <= x*n/2 x is the number of searches; n the input size; b the base of the logarithm for the sort, depending on the partitioning you use.
When this statement evaluates to true, you should switch methods from linear search to sort and search.
Generally speaking, a linear search through an unordered array will take n/2 steps on average, though this average will only play a big role once x approaches n. If you want to stick with big Omicron or big Theta notation then you can omit the /2 in the above.
Assuming n elements and m searches, with crude approximations
the cost of the sort will be C0.n.log n,
the cost of the m binary searches C1.m.log n,
the cost of the m linear searches C2.m.n,
with C2 ~ C1 < C0.
Now you compare
C0.n.log n + C1.m.log n vs. C2.m.n
or
C0.n.log n / (C2.n - C1.log n) vs. m
For reasonably large n, the breakeven point is about C0.log n / C2.
For instance, taking C0 / C2 = 5, n = 1000000 gives m = 100.
You should plot the complexities of both operations.
Linear search: O(n)
Sort and binary search: O(nlogn + logn)
In the plot, you will see for which values of n it makes sense to choose the one approach over the other.
This actually turned into an interesting question for me as I looked into the expected runtime of a quicksort-like algorithm when the expected split at each level is not 50/50.
the first question I wanted to answer was for random data, what is the average split at each level. It surely must be greater than 50% (for the larger subdivision). Well, given an array of size N of random values, the smallest value has a subdivision of (1, N-1), the second smallest value has a subdivision of (2, N-2) and etc. I put this in a quick script:
split = 0
for x in range(10000):
split += float(max(x, 10000 - x)) / 10000
split /= 10000
print split
And got exactly 0.75 as an answer. I'm sure I could show that this is always the exact answer, but I wanted to move on to the harder part.
Now, let's assume that even 25/75 split follows an nlogn progression for some unknown logarithm base. That means that num_comparisons(n) = n * log_b(n) and the question is to find b via statistical means (since I don't expect that model to be exact at every step). We can do this with a clever application of least-squares fitting after we use a logarithm identity to get:
C(n) = n * log(n) / log(b)
where now the logarithm can have any base, as long as log(n) and log(b) use the same base. This is a linear equation just waiting for some data! So I wrote another script to generate an array of xs and filled it with C(n) and ys and filled it with n*log(n) and used numpy to tell me the slope of that least squares fit, which I expect to equal 1 / log(b). I ran the script and got b inside of [2.16, 2.3] depending on how high I set n to (I varied n from 100 to 100'000'000). The fact that b seems to vary depending on n shows that my model isn't exact, but I think that's okay for this example.
To actually answer your question now, with these assumptions, we can solve for the cutoff point of when: N * n/2 = n*log_2.3(n) + N * log_2.3(n). I'm just assuming that the binary search will have the same logarithm base as the sorting method for a 25/75 split. Isolating N you get:
N = n*log_2.3(n) / (n/2 - log_2.3(n))
If your number of searches N exceeds the quantity on the RHS (where n is the size of the array in question) then it will be more efficient to sort once and use binary searches on that.
When analyzing algorithms, I don't usually have issues with for loop while I do have problems analyzing the running time of algorithms that involves while loop. I'll show an example to express my question.
The following one is a part of the coin-change algorithm (well-known algorithm for most of us) that I am studying right now:
counter = 0
s = 0
while(s <= n/100)
s = s+1
t1 = n- 100*s
h = 0
while(h <= t1/50)
h = h +1
t2 = t1 - 50*h
...
Can someone explain the best way of knowing the running time of such algorithms that have nested while loops?
The while loop is just another way of writing a for loop so assessing complexity should not be any different.
Here, the outer while loop runs n times in complexity world (proportional to n in real world) since s increases by 1 for each iteration and it runs until s reaches a value proportional to n.
The inner loop which I assume you are a little confused about, runs t1 times (again in complexity world) where t1 = n - 100s. Now you are thinking the algorithm is O(n^2) but t1 decreases in each iteration so the inner loop runs fewer number of times for each subsequent iteration and may be its not O(n^2).
t1 is different for each iteration so the entire set of iterations will run for: n - 100 + n - 200 + n - 300 + .... 0 times. Because the numbers of terms in this series is proportional to n, the summation will have a n squared term and for reporting complexity all lower order terms are ignored so you need not worry about what the rest of the numbers sum to. This algorithm is O(n^2).
The trick is to ignore constant and lower order terms at each step and it becomes easy !
Time complexity of Selection Sort(Worst case) using Pseudocode:
'Selection-Sort(A)
1 For j = 1 to (A.length - 1)
2 i = j
3 small = i
4 While i < A.length
5 if A[i] < A[small]
6 small = i
7 i = i + 1
8 swap A[small], A[j]
First step will occur n-1 times (n is length of array). So the second and third. I am stuck with 4th step whether it will occur n! times or something else.
The basic operation for this algorithm is the comparison at line 5, in the inner loop. Both loops are executed ≈ n times, i.e. the basic operation is executed n*n times ≈ n^2.
The time complexity for selection sort is O(n^2). It is same for worst best and average cases.
You should have look at the link below it gives a good rundown on selection sort.
https://www.khanacademy.org/computing/computer-science/algorithms/sorting-algorithms/a/analysis-of-selection-sort
Hope this helps.
edit:
When analyzing the time complexity of non recursive algorithms,
Decide on parameters indicating the input size
Identify the basic operation
Set up a sum indicating the number of times the basic operation is executed
Establish its order of growth
Give an asymptotic estimation
In this case the input size will be the size of the array, the basic operation is of comparison, the arithmetic sum would be,
Σ1≤ j ≤n-1 Σj≤ i ≤n or Σ0≤ j ≤n-2 Σj+1≤ i ≤n-1
This will evaluate to (n-1)(n/2) which is asymptotically O(n^2).
For more information i would recommend these two books,
Introduction to Design and Analysis of Algorithms - Anany Livitin
Introduction to Algorithms - Coreman
Hello I am weak in maths. but I an trying to solve the problem below. Am I doing it correctly>
Given: that A is big O, omega,or theta of B.
Question is:
A = n^3 + n * log(n);
B = n^3 + n^2 * log(n);
As an example, I take n=2.
A= 2^3+2log2 => 8.6
B= 2^3+2^2log2 => 9.2
A is lower bound of B..
I have other questions as well but i need to just confirm the method i am applying is correct or is there any other way to do so.
Am doing this right? Thanks in advance.
The idea behind the big O-notation is to compare the long term behaviour. Your idea (to insert n=2) reveals whether A or B is largest for small values of n. However O is all about large values. Part of the problem is to figure out what a large value is.
One way to get a feel of the problem is to make a table of A and B for larger and larger values of n:
A B
n=10
n=100
n=1000
n=10000
n=100000
n=1000000
The first entry in the table is A for n=10: A=10^3 + 10*log(10) = 1000+10*1 = 1010.
The next thing to do, is to draw graphs of A and B in the same coordinate system. Can you spot any long term relation between the two?
A n^3 + n *log(n) 1 + log(n)/n^2
--- = ------------------ = ----------------
B n^3 + n^2*log(n) 1 + log(n)/n
Since log(n)/n and also log(n)/n^2 have limit zero for n trending to infinity, the expressions 1+log(n)/n and 1+log(n)/n^2 in the canceled quotient A/B are bounded to both sides away from zero. For instance, there is a lower bound N such that both expressions fall into the interval [1/2,3/2] for all n > N. This means that all possibilities are true.
I was trying to find the complexities of an algorithm via different approaches. Mathematically I came across one O(m+n) and another O(mn) approach. However I am unable to grasp or say visualize this. It's not like I look at them and get the "Ahh! That's what's going on" feeling! Can someone explain this using their own examples or any other tool?
O(m+n) example:
for(int i = 0, i < m, i++)
//code
for(int j = 0, j < n, j++)
//code
m iterations of code happen. Then n iterations of code happens.
O(mn) example:
for(int i = 0, i < m, i++)
for(int j = 0, j < n, j++)
//code
For every iteration of m, we have n iterations of code. Imagine iterating over a non-square 2D array.
m and n do not necessarily equal the same value. If they did equal the same value, then for O(m+n):
O(m+n) => O(m+m) => O(2m) => O(m)
I'd recommend looking at this question/answer in order to understand that last transition.
And for O(mn):
O(mn) => O(mm) => O(m^2)
My recommendation for finding intuition is thought experiments as follows:
First, realize that m and n are two different measurements of the input. They might be the lengths of two input streams, the lengths of sides of a matrix, or the counts of two different attributes of the same data structure, such as edge and node count of the same graph, or any similar measures.
The intuition is that big-O expresses a bound on the true run time (or some other aspect such as comparison count or space needed) of an algorithm in terms of a simple function - call that R(m, n) - multiplied by some arbitrary constant. We ignore the constant factors and think of all algorithms bounded by the same R as a family by calling their run times O( R(m, n) ).
Consequently, big O(m + n) says that the true run time is bounded by some function R(m,n) = C(m + n) for suitably big m and n. For the graph example, this says that the actual run time of the algorithm will be bounded by a multiple of the sum of the number of vertices and edges.
You can think of the bounding function as a graph in 3d with axes m, n, and R(m,n). Or you can think of charts:
R(m,n) = m + n
--------------
m= 1 2 3 4
n=1 1 2 3 4
2 3 4 5 6
3 4 5 6 7
4 5 6 7 8
For R(m,n) = mn, you have
R(m,n) = mn
--------------
m= 1 2 3 4
n=1 1 2 3 4
2 2 4 6 8
3 3 6 9 12
4 4 8 12 16
As a 3d graph, the first function is a plane and the second is a much faster-growing function at almost all points. This means that if m and n grow large enough, an O(mn) bound will ultimately be larger (corresponding to a potentially slower program) than an O(m+n) because the constants become insignificant.
For an example of the cost of rapid growth, suppose an O(m+n) algorithm has an extra constant factor of 3 in its runtime bound (making it potentially very slow on small inputs compared to both algorithms above):
R(m,n) = 3(m + n)
--------------
m= 1 2 3 4
n=1 3 9 12 15
2 9 12 15 18
3 12 15 18 21
4 15 18 21 24
So the the O(m + n) looks like it's bound is less constrained than the O(mn) one in the chart above. But look at the case m=n=100. Here bound on the O(m + n) algorithm is 3(m + n) = 600. But the O(mn) algorithm with the small constant has bound mn = 10000. Clearly you want the first if m and n are large.
#Anonymous raised a fine point on the initial version of this article, which confused big-O and big-Theta. Big-O only deals with bounds or upper limits on the quantity being measured. For example, this means that every O(n) algorithm is also O(n log n) and O(n^2). If the real run time is bounded by the slower-growing function, it is also bounded by all faster-growing ones.
Yet it is quite common for people to say "this algorithms is O(n)" while meaning that the bound is tight. That is, that Cn is an upper bound on the run time for some constant C and Dn is also a lower bound for some other constant D (and suitably large n). Such a tight bound is properly stated as Theta(n), not O(n). The run time of a Theta(R(m, n)) algorithm is (roughly speaking) proportional to R(m, n).
I'll add finally that there are many cases where you can't ignore constants. There exist lots of algorithms in the literature that are asymptotically "faster" than those in common use, but have constants so large that for practical problem sizes they are always too slow. Computational geometry has many examples. Radix 2 sort is another. It's Theta(n), but in practice a good quicksort (Theta(n log n) average case) will beat it on arrays of size up to at least 10^8 integers.
O(m+n) is much (an order of magnitude) faster than O(mn).
The O(m+n) algorithm could be one that iterates 2 sets and does a constant time (O(1)) operation on each element.
The O(mn) algorithm could be one that iterates the first set and does a linear search (O(n)) for the matching element in the second set.
The O(mn) algorithm is probably what professors would call The Naive Approach