I'm currently doing an assignment that requires us to discuss time complexities of different algorithms.
Specifically sum1 and sum2
def sum1(a):
"""Return the sum of the elements in the list a."""
n = len(a)
if n == 0:
return 0
if n == 1:
return a[0]
return sum1(a[:n/2]) + sum1(a[n/2:])
def sum2(a):
"""Return the sum of the elements in the list a."""
return _sum(a, 0, len(a)-1)
def _sum(a, i, j):
"""Return the sum of the elements from a[i] to a[j]."""
if i > j:
return 0
if i == j:
return a[i]
mid = (i+j)/2
return _sum(a, i, mid) + _sum(a, mid+1, j)
Using the Master theorem, my best guess for both of theese are
T(n) = 2*T(n/2)
which accoring to Wikipedia should equate to O(n) if I haven't made any mistakes in my assumptions, however when I do a benchmark with different arrays of length N with random integers in the range 1 to 100, I get the following result.
I've tried running the benchmark a multiple of times and I get the same result each time. sum2 seems to be twice as fast as sum1 which baffles me since they should make the same amount of operations. (?).
My question is, are these algorthim both linear and if so why do their run time vary.
If it does matter, I'm running these tests on Python 2.7.14.
sum1 looks like O(n) on the surface, but for sum1 T(n) is actually 2T(n/2) + 2*n/2. This is because of the list slicing operations which themselves are O(n). Using the master theorem, the complexity becomes O(n log n) which causes the difference.
Thus, for sum1, the time taken t1 = k1 * n log n. For sum2, time taken t2 = k2 * n.
Since you are plotting a time vs log n graph, let x = log n. Then,
t1 = k1 * x * 10^x
t2 = k2 * 10^x
With suitable values for k1 and k2, you get a graph very similar to yours. From your data, when x = 6, 0.6 ~ k1 * 6 * 10^6 or k1 ~ 10^(-7) and 0.3 ~ k2 * 10^6 or k2 = 3 * 10^(-7).
Your graph has log10(N) on the x-axis, which means that the right-most data points are for an N value that's ten times the previous ones. And, indeed, they take roughly ten times as long. So that's a linear progression, as you expect.
Related
It is well known that Pascal's identity can be used to encode a combination of k elements out of n into a number from 0 to (n \choose k) - 1 (let's call this number a combination index) using a combinatorial number system. Assuming constant time for arithmetic operations, this algorithm takes O(n) time.†
I have an application where k ≪ n and an algorithm in O(n) time is infeasible. Is there an algorithm to bijectively assign a number between 0 and (n \choose k) - 1 to a combination of k elements out of n whose runtime is of order O(k) or similar? The algorithm does not need to compute the same mapping as the combinatorial number system, however, the inverse needs to be computable in a similar time complexity.
† More specifically, the algorithm computing the combination from the combination index runs in O(n) time. Computing the combination index from the combination works in O(k) time if you pre-compute the binomial coefficients.
Description of a comment.
For given combinatorial index (N), to find k'th digit it is needed to find c_k such that (c_k \choose k) <= N and ((c_k+1) \choose k) > N.
Set P(i,k) = i!/(i-k)!.
P(i, k) = i * (i-1) * ... * (i-k+1)
substitute x = i - (k-1)/2
= (x+(k-1)/2) * (x+(k-1)/2-1) * ... * (x-(k-1)/2+1) * (x-(k-1)/2)
= (x^2 - ((k-1)/2)^2) * (x^2 - ((k-1)/2-1)^2) * ...
= x^k - sum(((k-2i-1)/2)^2))*x^(k-2) + O(x^(k-4))
= x^k - O(x^(k-2))
P(i, k) = (i - (k-1)/2)^k - O(i^(k-2))
From above inequality:
(c_k \choose k) <= N
P(c_k, k) <= N * k!
c_k ~= (N * k!)^(1/k) + (k-1)/2
I am not sure how large is O(c_k^(k-2)) part. I suppose it doesn't influence too much. If it is of order (c_k+1)/(c_k-k+1) than approximation is very good. That is due:
((c_k+1) \choose k) = (c_k \choose k) * (c_k + 1) / (c_k - k + 1)
I would try algorithm something like:
For given k
Precalculate k!
For given N
For i in (k, ..., 0)
Calculate c_i with (N * i!)^(1/i) + (i-1)/2
(*) Check is P(c_i, k) <=> N * i!
If smaller check c_i+1
If larger check c_i-1
Repeat (*) until found P(c_i, i) <= N * i! < P(c_i+1, i)
N = N - P(c_i, i)
If approximation is good, number of steps << k, than finding one digit is O(k).
Pseudo code :
s <- 0
for i=1 to n do
if A[i]=1 then
for j=1 to n do
{constant number of elementary operations}
endfor
else
s <- s + a[i]
endif
endfor
where A[i] is an array of n integers, each of which is a random value between 1 and 6.
I'm at loss here...picking from my notes and some other sources online, I get
T(n) = C1(N) + C2 + C3(N) + C4(N) + C5
where C1(N) and C3(N) = for loops, and C4(N) = constant number of elementary operations. Though I have a strong feeling that I'm wrong.
You are looping from 1..n, and each loop ALSO loops from 1..n (in the worst case). O(n^2) right?
Put another way: You shouldn't be adding C3(N). That would be the case if you had two independent for loops from 1..n. But you have a nested loop. You will run the inner loop N times, and the outer loop N times. N*N = O(n^2).
Let's think for a second about what this algorithm does.
You are basically looping through every element in the array at least once (outer for, the one with i). Sometimes, if the value A[i] is 1, you are also looping again through the whole loop with the j for.
In your worst case scenario, you are running against an array of all 1's.
In that case, your complexity is:
time to init s
n * (time to test a[i] == 1 + n * time of {constant ...})
Which means: T = T(s) + n * (T(test) + n * T(const)) = n^2 * T(const) + n*T(test) + T(s).
Asymptotically, this is a O(n^2).
But this was a worst-case analysis (the one you should perform most of the times). What if we want an average case analysis?
In that case, assuming an uniform distribution of values in A, you are going to access the for loop of j, on average, 1/6 of the times.
So you'd get:
- time to init s
- n * (time to test a[i] == 1 + 1/6 * n * time of {constant ...} + 5/6 * T(increment s)
Which means: T = T(s) + n * (T(test) + 1/6 * n * T(const) + 5/6 * T(inc s)) = 1/6* n^2 * T(const) + n * (5/6 * T(inc s) + T(test)) + T(s).
Again, asymptotically this is still O(n^2), but according to the value of T(inc s) this could be larger or lower than the other case.
Fun exercise: can you estimate the expected average run time for a generic distribution of values in A, instead of an uniform one?
The following algorithm returns the previous larger element of an array. It is from page 11 of these notes.
// Input: An array of numeric values a[1..n]
// Returns: An array p[1..n] where p[i] contains the index of the previous
// larger element to a[i], or 0 if no such element exists.
previousLarger(a[1..n])
for (i = 1 to n)
j = i-1;
while (j > 0 and a[j] <= a[i]) j--;
p[i] = j;
return p
My homework question is: Given input sequence {a1,...,an} is a random permutation of the set {1,...,n}, what is the expected running time?
I think this requires some sort of probabilistic analysis, but I need some hints since I have only done worst-case analysis in the past. I'm trying to find a formula for the cost of the j-loop for a given i (1 + the number of times we do operation j--), then sum that formula up from 1 to n.
What does "expected" mean? I don't really know how to interpret this.
Building on #Heuster's answer:
1) You know that the answer is between O(n) and O(n^2). This is just to check the final result.
2) The expected number of steps for element i would indeed be:
sum_{k=1}^i 1 / (k+1)
= O(log i)
3) You have to sum all those number over i. This gives you:
sum_{i=1}^n O(log i)
= O(n log n)
What I did is not rigorous at all but you can prove derive it. O(n log n) is between O(n) and O(n^2) so it seems a good candidate :)
For and arbitrary index i, what is the chance that a[i-1] > a[i] (in other words, the inner while loop will take one step)? That one is easy: all elements in a are different, so P(a[i-1] > a[i]) = P(a[i] > a[i-1]) = 1/2.
Now, look at the case that the inner while loop would need to take two steps. That is, a[i-2] > a[i] > a[i-1]. This is exactly one of the 6 permutations of 3 elements, so the chance is 1 / 3! = 1 / 6.
Let's generalize this and assume that the inner while loop would need to take k steps. We consider the sublist a[i-k], a[i-k+1], ..., a[i]. We know that a[i-k] is the maximum element of this sublist and a[i] the second largest (otherwise, the inner loop would have stopped sooner). The order of the elements in between is irrelevant. The chance that we take k steps is thus 1 / (k + 1) * 1 / k = 1 / (k * (k + 1)). Note that this indeed degeneralizes to 1/2 for k = 1 and 1/6 for k = 2.
The chance that no element before a[i] is larger is simply 1 / i (a[i] is the maximum element in that sublist). In that case, the inner loop would need i steps.
The expected number of steps for element i would be (sum of probability times value):
Sum[{k, 1, i} k * 1 / ((k * (k + 1))] + i / i
= Sum[{k, 1, i} 1 / (k + 1)] + 1
= H_{i+1}
where H_{i} is the ith harmonic number, which is the discrete variant of log i. That is, the number of steps for element i is Θ(i).
What is remaining now is sum over all i to find the expected running time. With the exact value (H_{i+1}) this doesn't lead to a nice expression, see Wolfram Alpha.
The standard way to proceed, however, is to continue with the approximated log i. Clearly, log 0 + log 1 + ... + log n-1 is less than n log n. Now, consider the last half of the sum:
log n/2 + log n/2+1 + ... + log n-1 > n/2 log n/2
= n/2 (log n - log 2)
= Ω(n log n)
Therefore, the expected running time is Θ(n log n).
Consider the following randomized search algorithm on a sorted array a of length n (in increasing order). x can be any element of the array.
size_t randomized_search(value_t a[], size_t n, value_t x)
size_t l = 0;
size_t r = n - 1;
while (true) {
size_t j = rand_between(l, r);
if (a[j] == x) return j;
if (a[j] < x) l = j + 1;
if (a[j] > x) r = j - 1;
}
}
What is the expectation value of the Big Theta complexity (bounded both below and above) of this function when x is selected randomly from a?
Although this seems to be log(n), I carried out an experiment with instruction count, and found out that the result grows a little faster than log(n) (according to my data, even (log(n))^1.1 better fit the result).
Someone told me that this algorithm has an exact big theta complexity (so obviously log(n)^1.1 is not the answer). So, could you please give the time complexity along with your approach to prove it? Thanks.
Update: the data from my experiment
log(n) fit result by mathematica:
log(n)^1.1 fit result:
If you're willing to switch to counting three-way compares, I can tell you the exact complexity.
Suppose that the key is at position i, and I want to know the expected number of compares with position j. I claim that position j is examined if and only if it's the first position between i and j inclusive to be examined. Since the pivot element is selected uniformly at random each time, this happens with probability 1/(|i - j| + 1).
The total complexity is the expectation over i <- {1, ..., n} of sum_{j=1}^n 1/(|i - j| + 1), which is
sum_{i=1}^n 1/n sum_{j=1}^n 1/(|i - j| + 1)
= 1/n sum_{i=1}^n (sum_{j=1}^i 1/(i - j + 1) + sum_{j=i+1}^n 1/(j - i + 1))
= 1/n sum_{i=1}^n (H(i) + H(n + 1 - i) - 1)
= 1/n sum_{i=1}^n H(i) + 1/n sum_{i=1}^n H(n + 1 - i) - 1
= 1/n sum_{i=1}^n H(i) + 1/n sum_{k=1}^n H(k) - 1 (k = n + 1 - i)
= 2 H(n + 1) - 3 + 2 H(n + 1)/n - 2/n
= 2 H(n + 1) - 3 + O(log n / n)
= 2 log n + O(1)
= Theta(log n).
(log means natural log here.) Note the -3 in the low order terms. This makes it look like the number of compares is growing faster than logarithmic at the beginning, but the asymptotic behavior dictates that it levels off. Try excluding small n and refitting your curves.
Assuming rand_between to implement sampling from a uniform probability distribution in constant time, the expected running time of this algorithm is Θ(lg n). Informal sketch of a proof: the expected value of rand_between(l, r) is (l+r)/2, the midpoint between them. So each iteration is expected to skip half of the array (assuming the size is a power of two), just like a single iteration of binary search would.
More formally, borrowing from an analysis of quickselect, observe that when you pick a random midpoint, half of the time it will be between ¼n and ¾n. Neither the left nor the right subarray has more than ¾n elements. The other half of the time, neither has more than n elements (obviously). That leads to a recurrence relation
T(n) = ½T(¾n) + ½T(n) + f(n)
where f(n) is the amount of work in each iteration. Subtracting ½T(n) from both sides, then doubling both sides, we have
½T(n) = ½T(¾n) + f(n)
T(n) = T(¾n) + 2f(n)
Now, since 2f(n) = Θ(1) = Θ(n ᶜ log⁰ n) where c = log(1) = 0, it follows by the master theorem that T(n) = Θ(n⁰ lg n) = Θ(lg n).
I came across this problem of finding said probability and my first attempt was to come up with following algorithm: I am counting number of pairs which are relatively prime.
int rel = 0
int total = n * (n - 1) / 2
for i in [1, n)
for j in [i+1, n)
if gcd(i, j) == 1
++rel;
return rel / total
which is O(n^2).
Here is my attempt to reducing complexity:
Observation (1): 1 is relatively prime to [2, n] so n - 1 pairs are trivial.
Observation (2): 2 is not relatively prime to even numbers in the range [4, n] so remaining odd numbers are relatively prime to 2, so
#Relatively prime pairs = (n / 2) if n is even
= (n / 2 - 1) if n is odd.
So my improved algorithm would be:
int total = n * (n - 1) / 2
int rel = 0
if (n % 2) // n is odd
rel = (n - 1) + n / 2 - 1
else // n is even
rel = (n - 1) + n / 2
for i in [3, n)
for j in [i+1, n)
if gcd(i, j) == 1
++rel;
return rel / total
With this approach I could reduce two loops, but worst case time complexity is still O(n^2).
Question: My question is can we exploit any mathematical properties other than above to find the desired probability in linear time?
Thanks.
You'll need to calculate the Euler's Totient Function for all integers from 1 to n. Euler's totient or phi function, φ(n), is a arithmetical function that counts the number of positive integers less than or equal to n that are relatively prime to n.
To calculate the function efficiently, you can use a modified version of Sieve of Eratosthenes.
Here is a sample C++ code -
#include <stdio.h>
#define MAXN 10000000
int phi[MAXN+1];
bool isPrime[MAXN+1];
void calculate_phi() {
int i,j;
for(i = 1; i <= MAXN; i++) {
phi[i] = i;
isPrime[i] = true;
}
for(i = 2; i <= MAXN; i++) if(isPrime[i]) {
for(j = i+i; j <= MAXN; j+=i) {
isPrime[j] = false;
phi[j] = (phi[j] / i) * (i-1);
}
}
for(i = 1; i <= MAXN; i++) {
if(phi[i] == i) phi[i]--;
}
}
int main() {
calculate_phi();
return 0;
}
It uses the Euler's Product Formula described on the Wikipedia page of Totient Function.
Calculating the complexity of this algorithm is a bit tricky, but it is much less than O(n^2). You can get results for n = 10^7 pretty quickly.
The number of integers in the range 0 .. n that are coprime to n is the Euler totient function of n. You are computing the sum of such values, e.g. called summatory totient function. Methods to compute this sum fast are for example
described here. You should easily get a method with a better than quadratic complexity,
depending on how fast you implement the totient function.
Even better are the references listed in the encyclopedia of integer sequences: http://oeis.org/A002088, though many of the references require some math skills.
Using these formulas you can even get an implementation that is sublinear.
For each prime p, probability of it dividing a randomly picked number between 1 and n is
[n / p] / n
([x] being the biggest integer not greater than x). If n is large, this is approximately 1/p.
The probability of it dividing two such randomly picked numbers is
([n / p] / n)2
Again, this is 1/p2 for large n.
Two numbers are coprime if no prime divides both, so the probability in question is the product
Πp is prime(1 - ([n / p] / n)2)
It is enough to calculate it for all primes less than or equal to n. As n goes to infinity, this product approaches 6/π2.
I'm not sure you can use the totient function directly, as described in the other answers.