I have these algorithm and I didn't find out which one takes more time.
O((n^2)*log(n))
O(n*(2^n))
I calculate the log of those but I can't understand which one takes more time.
log((n^2)*log(n)) = 2log(n)+log(log(n))
log(n*(2^n))=log(n)+n*log(2)
The second one, because:
2^n > n^2
and
n > log(n)
For a small range of values of n (roughly n = 5 (~4.738) to 26 (~25.783)) assuming natural logarithms, the first is larger than the second, but above that the second is always larger and becomes increasingly so as n increases.
Plotting it confirms this, here using Mathematica:
f1[n_] := Log[n^2]*Log[n]
f2[n_] := Log[n*(2^n)]
Plot[{f1[n], f2[n]}, {n, 1, 50}]
Not good enough at math to show proof, but simply filling in a value for "n" will give you a good indication. Let's take n = 100, base for log is 10.
# First algorithm
100^2 * log(100) = 10000 * 2 = 20000
# Second algorithm
100 * 2^100 = 100 * 1267650600228229401496703205376 = 126765060022822940149670320537600
I think it should be pretty obvious to always choose the first algorithm.
You can calculate, reduce, or simplify those expressions.
..but hold on a moment and pay attention, that there is a highest order operand 2n, which grows exponentially - way faster (for sufficiently large n-s) than any other of the operands you have - namely n2 or log2n.
Therefore, O((n*(2n)) would take way much more for sufficiently large inputs.
input 1 → 2 operations
input 2 → 8 operations
input 3 → 24 operations
input 4 → 64 operations
...
input 10 → 10240 operations
I think you can see the pattern.
Note, that in terms of asymptotic analysis, we are always interested in large inputs, otherwise, for 1, 2 or very small inputs behaviour will be different.
Related
I had this question on an exam,
I have a very slow computer which takes one second for binary search on an array with 1,000 (one thousand) elements. How long should I expect the computer to take for binary search on an array with 1,000,000 (one million) elements?
I have trouble understanding this, would it take longer to search through 1 million elements than 1,000 or does it depend on the computer?
Binary search has O(log(N)) time complexity, completion time is
t = c * log(N, 10)
In given case for N = 1000 we know t and thus we can find out c
1 = c * log(1000, 10)
1 = c * 3
c = 1/3
For N = 1000000 we have
t = 1 / 3 * log(1000000, N) =
= 1 / 3 * 6 =
= 2
So we can expect that binary search within 1000000 items will be completed in 2 seconds.
Please, note that O(log(N)) == O(log(N, m)) since log(N, m) == log(N, k) / log(m, k), which means that when working with O(log(N)) we are free to choose logarithm base. In our case (1000 and 1000000) it's convenient to use base 10 (or 1000)
First, a binary search requires the list or array to be sorted. And since you are dividing each list/sublist in half it will take log2(N) searches to find the correct item where log2 is log to the base 2.
So for 1_000_000 items it should only take about 20 comparisons to home in on the item. So it should be very quick (sans the time to sort the list to begin with).
1000 items would take about 10 comparisons. And one second, imo, is much to long to do such a simple search on any modern day processor.
It does depend on the computer and also depends on the size of the array. Since in this question the same computer is used, we can abstract the effects of the computer and focus on the sample size.
Binary search has logarithmic time complexity. If you take the difference between log(1_000) and log(1_000_000) you will see that you should expect double the time (2 seconds) to search in the 1 million elements array.
This is assuming worst case. For the average case, the calculation gets more complex, you can check this wikipedia page for a deeper analysis.
I try to understand a formula when we should use quicksort. For instance, we have an array with N = 1_000_000 elements. If we will search only once, we should use a simple linear search, but if we'll do it 10 times we should use sort array O(n log n). How can I detect threshold when and for which size of input array should I use sorting and after that use binary search?
You want to solve inequality that rougly might be described as
t * n > C * n * log(n) + t * log(n)
where t is number of checks and C is some constant for sort implementation (should be determined experimentally). When you evaluate this constant, you can solve inequality numerically (with uncertainty, of course)
Like you already pointed out, it depends on the number of searches you want to do. A good threshold can come out of the following statement:
n*log[b](n) + x*log[2](n) <= x*n/2 x is the number of searches; n the input size; b the base of the logarithm for the sort, depending on the partitioning you use.
When this statement evaluates to true, you should switch methods from linear search to sort and search.
Generally speaking, a linear search through an unordered array will take n/2 steps on average, though this average will only play a big role once x approaches n. If you want to stick with big Omicron or big Theta notation then you can omit the /2 in the above.
Assuming n elements and m searches, with crude approximations
the cost of the sort will be C0.n.log n,
the cost of the m binary searches C1.m.log n,
the cost of the m linear searches C2.m.n,
with C2 ~ C1 < C0.
Now you compare
C0.n.log n + C1.m.log n vs. C2.m.n
or
C0.n.log n / (C2.n - C1.log n) vs. m
For reasonably large n, the breakeven point is about C0.log n / C2.
For instance, taking C0 / C2 = 5, n = 1000000 gives m = 100.
You should plot the complexities of both operations.
Linear search: O(n)
Sort and binary search: O(nlogn + logn)
In the plot, you will see for which values of n it makes sense to choose the one approach over the other.
This actually turned into an interesting question for me as I looked into the expected runtime of a quicksort-like algorithm when the expected split at each level is not 50/50.
the first question I wanted to answer was for random data, what is the average split at each level. It surely must be greater than 50% (for the larger subdivision). Well, given an array of size N of random values, the smallest value has a subdivision of (1, N-1), the second smallest value has a subdivision of (2, N-2) and etc. I put this in a quick script:
split = 0
for x in range(10000):
split += float(max(x, 10000 - x)) / 10000
split /= 10000
print split
And got exactly 0.75 as an answer. I'm sure I could show that this is always the exact answer, but I wanted to move on to the harder part.
Now, let's assume that even 25/75 split follows an nlogn progression for some unknown logarithm base. That means that num_comparisons(n) = n * log_b(n) and the question is to find b via statistical means (since I don't expect that model to be exact at every step). We can do this with a clever application of least-squares fitting after we use a logarithm identity to get:
C(n) = n * log(n) / log(b)
where now the logarithm can have any base, as long as log(n) and log(b) use the same base. This is a linear equation just waiting for some data! So I wrote another script to generate an array of xs and filled it with C(n) and ys and filled it with n*log(n) and used numpy to tell me the slope of that least squares fit, which I expect to equal 1 / log(b). I ran the script and got b inside of [2.16, 2.3] depending on how high I set n to (I varied n from 100 to 100'000'000). The fact that b seems to vary depending on n shows that my model isn't exact, but I think that's okay for this example.
To actually answer your question now, with these assumptions, we can solve for the cutoff point of when: N * n/2 = n*log_2.3(n) + N * log_2.3(n). I'm just assuming that the binary search will have the same logarithm base as the sorting method for a 25/75 split. Isolating N you get:
N = n*log_2.3(n) / (n/2 - log_2.3(n))
If your number of searches N exceeds the quantity on the RHS (where n is the size of the array in question) then it will be more efficient to sort once and use binary searches on that.
I am looking for an explanation for this question since i'm studying for GRE:
An algorithm is run in 10 seconds for a size 50 entry. If the algorithm is quadratic, how long in seconds does it spend to approximately on the same computer if the input has size 100?
I can't see a relation between time and the input.
Considering: O(n^2) -> O(50^2) =! 10 (seconds)
Also, i would like to study more about this topic, so please add any source if you can.
Thanks.
Note that the terminology is sloppy, time complexity has no notion of time (yes, the name is deceiving).
Neglecting terms smaller than O(n2) since we are working under the big-O framework, a quadratic function can be expressed as
t = c * n2
Given the (t, n) pair with value (10, 50) we have
10 = c * 2500
c = 1/250 = 4/1000
Solving for (t, 100) we get
t = 4/1000 * 10000 = 40
There is a faster and more insightful way to solve this problem.
The trained eye can spot the answer as 40 immediately.
Consider this function power function:
t = c * nk
Now lets consider the inputs m0 and m1, with m1 = a * m0 being an (integer) multiple of m0.
Lets compare the respective t0, t1:
t0 = c * (m0)k
t1 = c * (m1)k = c * ak * (m0)k
So we see that for a polynomial time function t1/t0 = ak, or t1 = ak * t0.
The ratio between two output is the ratio between their inputs, to the k-th power.
For a quadratic function k = 2, thus the ration between two outputs is the square of the ratio between two inputs.
If the input doubles (ratio = 2) the output quadruples (ratio = 22 = 4).
If the input triples (ratio = 3) the output is nine-fold (ratio = 32 = 9).
A good mnemonic trick is to remember that the function from the inputs ratio to the outputs ratio is of the same kind of the given function.
We are given a quadratic function so that is the kind of relationship between the inputs and outputs ratios:
input output
ratio ratio
1 1
2 4
3 9
4 16
5 25
... ...
The problem tells you that the output doubles (from 50 to 100 entries) so the output must quadruple (from 10 to 40) as the function is quadratic.
As you can see all the data from the problem is used elegantly and without any hardcore computation.
Suggesting out-site sources is frowned upon but in this case I can't help but recomending the reading of:
Introduction to the Theory Computation - Michael Sipser
See if you can answer these questions:
If the input is now 150 entries, how much time does it take?
90
If the input is now 30 entries, how much time does it take?
90/25 ~ 4
If the input is now 100 entries but the program is run in a computer that is twice as fast, how much time does it take?
20
What size of the input is necessary to make the program run for at least 1000 seconds?
500
Given the time complexity you can't calculate the exact running time (in seconds) of the algorithm. However, it does give you a good idea of the growth rate of measured time.
In a linear time algorithm (O(n)), time is expected to increase linearly as a function of the input. For example, if 10,000 items take one second to process in some machine, then you should expect 50,000 items to take about 5 seconds, and so on. This isn't the case in a quadratic algorithm (O(n^2)), which "punishes" you more as the input size is larger; an increase of x2 in input size will result in x4 processing time, an increase x5 in input size will result in x25 processing time, etc (Just like the function F(x)=x^2 behaves).
You can use this post as an introduction, and this one as a more formal explanation.
The Big-O doesn't inform you about the exact correlation between entry size and the execution time, but instead it tells what's the approximate correlation between the entry size growth and the execution time growth. So: O(n) complexity means that the entry size and the execution time are in direct proportion (if the input is 3 times bigger, then the execution time will also be 3 times longer), O(n^2) means that if the entry size is 3 times bigger, then the execution time will be 9 times longer etc.
In your case, the initial execution time for a size 50 entry is 10 seconds. If the input changes to a size 100 entry, then by simply dividing we can tell that it's 100 / 50 = 2 times bigger. Knowing that the algorithm's Big-O is O(n^2), we can tell that the execution time will be 2^2 = 4 times longer. So if the initial execution time was 10 seconds, then the execution time for the bigger input will be approximately 10 seconds * 4 = 40 seconds.
Related SO question: Rough estimate of running time from Big O
Nice article about the big-o: https://rob-bell.net/2009/06/a-beginners-guide-to-big-o-notation/
I have 2 anagram detection functions; one uses sort and compare and the other keeps track of the number of occurences of each alphabetical character.
Assume here that the two strings passed to the functions are identical, the first randomly generated (unsorted) and the second = to the first, so that both functions execute "all the way" and return True.
According to my algorithmic analysis (obviously probably wrong...), the first function should be 2 * n log(n) + n, and the second function 2 + 2n + 2n + 26 = 28 + 5n.
def anag1(s1, s2):
s1 = list(s1)
s1.sort()
s2 = list(s2)
s2.sort()
for i in range(len(s1)):
if s1[i] != s2[i]:
return False
return True
def anag2(s1, s2):
l1 = [0] * 26
l2 = [0] * 26
for i in s1:
k = ord(i) % 26
l1[k] += 1
for i in s2:
k = ord(i) % 26
l2[k] += 1
return l1 == l2
For 2 strings of 100,000 characters, this should mean that the first function would be 3.4 million time units vs 400,000 time units for the second function.
In other words, the second function should be close to 8 times faster than the first.
However, when I time the execution of both function, the second one is not even twice as fast as the first. For 2 strings of 100,000 characters, the first function executes in 0.085s and the second in 0.055s.
What's going on?
Theoretical complexity may have little with actual time spent. Consider:
Various complexity of single operations (e.g. division vs addition)
Memory access (bad cache hit ratio due to frequent random access on huge arrays can slow it down to more than half)
Compiler / interpreter optimizations
etc.
And not every sort is O(n log (n))
Counting sort, Bucket Sort, Radix Sort are all O(n) or close to.
Edit:
I implemented both in Java on 100M number long array. It was 383 vs. 161 ms. On 10M the times were almost equal.
Your 100k long array is way too small to warm up optimizers.
Java uses DualPivotQuickSort, which runs almost O(n) on arrays with many duplicate values (small character cardinality). Your language may use something similar.
Not all primitive functions take the same amount of time. For example, your second method includes divisions, which often take more CPU cycles than other operations. While the first function is O(n log(n)) and the second is O(n), you don't know what the constant factor is.
What your analysis really indicates is that, if you now do it with million character strings, you should see that the difference in speeds widens.
Suppose an algorithm is known to be O(N2) and solving a problem of size M takes 5 minutes. About how long will it take to solve a problem of size 4M?
Is it as simple as ...
M=5min
4M=20min
?
Since Big O is just an approximation you can not compute the real time but yes you can have some estimation. In your case it would be
1 M ~ 5 min
4 M ~ 5 *(4*4) min ~ 80 min.
Note : I used symbol ~ to show approximation.
O(N^2) => problem with size N will take approximately N^2 time
M will take approximately M^2 time
O(M)~ O(1M)
=> 1^2*M^2
=> M^2
=> 5 min
O(4M) ~ (4M)^2
=> 4^2*M^2
=> 16*M^2
=> 16*5
=> 80 min
If the complexity is O(N^2), this means the time for a problem of size N is roughly k*N^2 for some fixed but unknown value of k.
If you represent the approximate time to run the algorithm on a problem of size N as T(N), then mathematically you have this:
T(N) = k*N^2
T(M) = k*M^2
T(M) = 5 minutes
T(4*M) = k*(4*M)^2
= 16*k*M^2
= 16*T(M)
= 80 minutes
In a nutshell, not necesarily.
When we say that a problem's time complexity is O(N2), what that means is that given a problem of size N, the time it takes to run conforms roughly to some equation of the form a + bN + cN2, where a, b, and c are unknown coefficients.
This does mean that eventually the N2 term will dominate the run-time. But eventually might be a long time away. There might be a large constant set-up time built in (that is, a in the formula above is big), such that 4 of the 5 minutes of your hypothetical scenario don't vary with problem size. In that case, perhaps a problem of size 4M might take less than twice as long to run.
Situations along these lines can happen frequently with algorithms that involve hashing (such as some associative array implementations), particularly if a slow hash function such as SHA2 is being used. Which is why for small collections of elements searching a simple array to see if it contains an element might be faster than searching a hash table, even though searching an array is O(N) and searching a hash set is O(1).
Yes, it is simple, but your calculation is wrong.
What you have calculated is a linear growth, e.g. something of growth of O(n) e.g. if some input takes five minutes, you double the size of your input, then time spend is twice that time. You state that your algorithm run in O(n^2) which is exponential growth.
So your calculation would look like this:
M^2 = 5 minutes <=>
M = sqrt(5) = 2.23607 (approx)
so
(4M)^2 = (4*2.23607)^2 = 80 minutes
Which is exponential growth.
This is also why you never talk about specific run times in computer science. Whether something takes 5 minutes or 5 hours is not interesting. What is interesting is what happens when we change the size of the input. Because when we implement algorithms we want something that runs faster, no matter what computer is used for testing when the size of the input moves towards infinite.
Your guess is correct in case you have O(n), but we have O(n^2) which means you need to square the constant
T(M) = (M)^2 = M^2 = 5 minutes
T(4M) = (4 * M)^2 = 4^2 * M^2 = 16 * M^2
Substitute: M^2 = 5 minutes
T(4M) = 16 * M^2 = 16 * 5 minutes
T(4M) = 80 minutes