I was watching a lecture about program efficiency where the professor said:
"If I measure running time, it will certainly vary as the algorithm change. (...) But one of the problems is that it will also vary as a function of the implementation (...) If I use a loop that's got a couple of more steps inside of it in one algorithm than another, it's going to change the time."
I am having a hard time wrapping my head around the implementation's influence.
So my question is: Why can't we consider those extra loop steps inside of one algorithm, when compared to the other, simply something that is necessary for it to run and that is also a part of the algorithm's efficiency? Or did I completely miss the point here?
Thank you!
They are pointing out the difference between "algorithm" and "specific code written in a programming language". "Algorithm" is somewhat of a vague term and "algorithms" are often described in pseudo-code that can be either very detailed, or not detailed at all.
For instance, consider the following algorithm, to test whether a number n is prime or not:
If any number d between 2 and the square root of n divides n:
Return False (n is not prime)
Else:
Return True (n is prime)
How exactly do you loop over the numbers between 2 and the square root? You could do:
// IMPLEMENTATION A
bool is_prime(int n)
{
int s = sqrt(n);
for (int d = 2; d <= s; d++)
{
if (n % d == 0)
return false;
}
return true;
}
or:
// IMPLEMENTATION B
bool is_prime(int n)
{
for (int d = 2; d * d <= n; d++)
{
if (n % d == 0)
return false;
}
return true;
}
Those two codes both implement the algorithm that I described. However, they might not have exactly the same runtime, since the former requires computing sqrt(n) once, but the latter requires computing d * d at every iteration in the loop.
Computer scientists want to be able to discuss the complexity of the algorithm that I described above in pseudo-code. And they don't want someone to give the boring answer "Sorry, the complexity of that algorithm is impossible to calculate, because we don't know if it's implementation A or implementation B".
Related
Despite reading some previous questions here on stackoverflow and watching a few videos including this
one, time and space complexity are going straight over my head. I need to find the time and space complexity of this algorithm
public static int aPowB(int a, int b){
if(b == 0){
return 1;
}
int halfResult = aPowB(a, b/2);
if(b%2 == 0){
return halfResult * halfResult;
}
return a * halfResult * halfResult;
}
An explanation of the answer would be appreciated so I can try to understand. Thank you.
First of all, the inputs are a and b, so we can expect the time/space complexity to be dependent on this two parameters.
With recursive algorithms, always try to write down the recurrence relation for the time complexity T first. Here it's
T(a, 0) = O(1) // base case
T(a, b) = T(a, b/2) + O(1) // recursive call + some O(1) stuff at the end
This equation is one of the standard ones that you should just know by heart, so we can immediately give the solution
T(a, b) = O(log b)
(If you don't know the solution by hard, just ask yourself how many times you can divide b by 2 until you hit 0.)
The space complexity is also O(log b) because that's the depth of the recursion stack.
Hello StackOverflow community!
I had this question in my mind from so many days and finally have decided to get it sorted out. So, given a algorithm or say a function which implements some non-standard algorithm in your daily coding activity, how do you go about analyzing the rum time complexity?
Ok let me be more specific. Suppose you are solving this problem,
Given a NxN matrix consisting of positive integers, find the longest increasing sequence in it. You may only traverse in up, down, left or right directions but not diagonally.
Eg: If the matrix is
[ [9,9,4],
[6,6,8],
[2,1,1] ].
the algorithm must return 4
(The sequence being 1->2->6->9)
So yeah, looks like I have to use DFS. I get this part. I have done my Algorithms course back in Uni and can work my way around such questions. So, I come up with this solution say,
class Solution
{
public int longestIncreasingPathStarting(int[][] matrix, int i, int j)
{
int localMax = 1;
int[][] offsets = {{0,1}, {0,-1}, {1,0}, {-1,0}};
for (int[] offset: offsets)
{
int x = i + offset[0];
int y = j + offset[1];
if (x < 0 || x >= matrix.length || y < 0 || y >= matrix[i].length || matrix[x][y] <= matrix[i][j])
continue;
localMax = Math.max(localMax, 1+longestIncreasingPathStarting(matrix, x, y));
}
return localMax;
}
public int longestIncreasingPath(int[][] matrix)
{
if (matrix.length == 0)
return 0;
int maxLen = 0;
for (int i = 0; i < matrix.length; ++i)
{
for (int j = 0; j < matrix[i].length; ++j)
{
maxLen = Math.max(maxLen, longestIncreasingPathStarting(matrix, i, j));
}
}
return maxLen;
}
}
Inefficient, I know, but I wrote it this way on purpose! Anyways my question is, how do you go about analyzing the run time of longestIncreasingPath(matrix) function?
I can understand the analysis they teach us in a Algos course, you know the standard MergeSort, QuickSort analysis etc. but unfortunately and I hate to say this, that did not prepare me to apply it in my day-day coding job. I want to do it now, and hence would like to start it by analyzing such functions.
Can someone help me out here and describe the steps one would take to analyze the runtime of the above function? That would greatly help me. Thanks in advance, Cheers!
For day to day work eye-balling things usually works well.
In this case you will try to go in every direction recursively. So a really bad example comes to mind like: [[1,2,3], [2,3,4], [3,4,5]] so that you have two options from most cells. I happen to know that this will be O((2*n) ! / (n!*n!)) steps, but another good guess would be O(2^N). Now that you have an example where you know or can compute more easily the complexity, the overall complexity has to be at least that.
Usually, it doesn't really matter which one it is exactly since for both O(N!) and O(2^N) the run-time grows very fast and should only work fast for up to around 10-20 maybe a bit more if you are willing to wait. You would not run this algorithm for N ~= 1000, you would need something polynomial. So an rough estimate that you have a exponential solution would be enough to make a decision.
So in general to get an idea of the complexity, try to relate your solution to other algorithms where you know the complexity already or figure out a worst case scenario for the algorithm where it's easier to judge the complexity. Even if you are slightly off it might still help you make a decision.
If you need to compare algorithms of more similar complexity (ie. O(NlogN) vs O(N^2) for N~=100) you should implement both and benchmark since the constant factor might be the leading contributor to the run-time.
I’m studying algorithm - time complexity and recursion.
I’m actually ok with solving recursion, cuz it’s simple math. But code part is the problem.
For example, This is the problem I’ve brought :
https://brilliant.org/practice/big-o-notation/?problem=complexityrun-time-analysis-2-2
public int P(int x , int n){
if (n == 0){
return 1;
}
if (n % 2 == 1){
int y = P(x, (n - 1) / 2);
return x * y * y;
}
else{
int y = P(x, n / 2);
return y * y;
}
}
It is a simple power function. T(n)=O(g(n)) is running time of this function for large, and I have to find it.
The solution says,
“When the power is odd an extra multiplication operation is performed. To work out time complexity, let us first look at the worst scenario, meaning let us assume that one additional multiplication operation is needed.”
However, I do not understand the next part, the solution says that :
Recursion relation is
T(n) = T(n/2) + 3, T(1)=1
1) Why is the constant part 3?
if (n % 2 == 1){
int y = P(x, (n - 1) / 2);
return x * y * y;
}
2) I actually don’t get exactly why T(1)=1 also.
I’m puzzled with.. which operations should we consider while calculating time complexity?
For example, T(1)=1 part must be related with
if (n == 0){
return 1;
}
if (n % 2 == 1){
int y = P(x, (n - 1) / 2);
return x * y * y;
}
This part, and I want to ask whether T(1)=1 comes from if statement/assign statement/return statement..
I understand afterwards, solving the recursion relation above, but I’m stuck with the recursion relation itself.
Please help me algo gurus..
which operations should we consider while calculating time complexity?
The answer will disappoint you a bit: it doesn't matter what operations you count. That's why we use big-Oh in analysing algorithms and expressing their time/memory requirements. It is an asymptotic notation that describes what happens to the algorithm for large values of n. By the definition of Big-Oh, we can say that both 1/2n^2 and 10n^2+6n+100 are O(n^2), even if they are not the same function. Counting all the operations, will just increase some constant factors, and that's why it doesn't really matter which ones you count.
By the above, the constants are simply O(1). This disregards details, since both 10 and 10000 are O(1), for example.
One could argue that specifying the exact number of operations in the expression T(n) = T(n/2) + 3 is not very correct, since there is no definition for what an operation is, and moreover the same operation might take a different amount of time on different computers, so exactly counting the number of operations is a bit meaningless at best and simply wrong at worst. A better way of saying it is T(n) = T(n/2) + O(1).
T(1)=1 represents the base case, which is solved in constant time (read: a constant number of operations at each time). Again, a better (more formal) way of saying that is T(1)=O(1).
I wrote this prime factorization function, can someone explain the runtime to me? It seems fast to me as it continuously decomposes a number into primes without having to check if the factors are prime and runs from 2 to the number in the worst case.
I know that no functions yet can factor primes in polynomial time. Also, how does the run time relate asymptotically to factoring large primes?
function getPrimeFactors(num) {
var factors = [];
for (var i = 2; i <= num; i++) {
if (num % i === 0) {
num = num / i;
factors.push(i);
i--;
}
}
return factors;
}
In your example, if num is prime then it would take exactly num - 1 steps. This would mean that the algorithm's runtime is O(num) (where O stands for a pessimistic case). But in case of algorithm that operate on numbers things get a little bit more tricky (thanks for noticing thegreatcontini and Chris)! We always describe complexity as a function of input size. In this case the input is a number num and it is represented with log(num) bits. So the input size is of log(num). Because num = 2 ^ (log(num)) then your algorithm is of complexity O(2^k) where k = log(num) - size of your input.
This is what makes this problem hard - input is very, very small and any polynomial from num leads to exponential algorithm ...
On a side note #rici is right, you need to check only up to sqrt(num), thus easily reducing the runtime to O(sqrt(num)) or more correctly O(sqrt(2) ^ k).
I know how to find recurrence relation from simple recursive algorithms.
For e.g.
QuickSort(A,LB, UB){
key = A[LB]
i = LB
j = UB + 1
do{
do{
i = i + 1
}while(A[i] < key)
do{
j = j - 1
}while(A[j] > key)
if(i < j){
swap(A[i]<->A[j])
}while(i <= j)
swap(key<->A[j]
QuickSort(A,LB,J-1)
QuickSort(A,J+1,UB)
}
T(n) = T(n - a) + T(a) + n
In the above recursive algorithm it was quite easy to understand how the input size is reducing after each recursive call. But how to find recurrence relation for any algorithm in general, which is not recursive but might also be iterative. So i started learning how to convert iterative algorithm to recursive just to make it easy to find recurrence relation.
I found this link http://refactoring.com/catalog/replaceIterationWithRecursion.html.
I used to convert my linear search algorithm to recursive.
LinearSearch(A,x,LB,UB){
PTR = LB
while(A[PTR]!=x && PTR<=UB){
if(PTR==UB+1){
print("Element does not exist")
}
else{
print("location"+PTR)
}
}
}
got converted to
LinearSearch(A,x,LB,UB){
PTR=LB
print("Location"+search(A,PTR,UB,x))
}
search(A,PTR,UB,x){
if(A[PTR]!=x && PTR<=UB){
if(PTR==UB+1){
return -1
}
else{
return search(A,PTR+1,UB,x)
}
}
else{
return PTR
}
}
This gives the recurrence relation to be T(n) = T(n-1) + 1
But i was wondering is this the right approach to find recurrence relation for any algorithm?
Plus i don't know how to find recurrence relation for algorithms where more than one parameter is increasing or decreasing.
e.g.
unsigned greatest_common_divisor (const unsigned a, const unsigned b)
{
if (a > b)
{
return greatest_common_divisor(a-b, b);
}
else if (b > a)
{
return greatest_common_divisor(a, b-a);
}
else // a == b
{
return a;
}
}
First of all, algorithms are very flexible so you should not expect to have a simple rule that covers all of them.
That said, one thing that I think will be helpful for you is to pay more attention to the structure of the input you pass to your algorithm than to the algorithm yourself. For example, consider that QuickSort you showed in your post. If you glance at those nested do-whiles you are probably going to guess its O(N^2) when in reality its O(N). The real answer is easier to find by looking at the inputs: i always increases and j always decreases and when they finaly meet each other, each of the N indices of the array will have been visited exactly once.
Plus I don't know how to find recurrence relation for algorithms where more than one parameter is increasing or decreasing.
Well, those algorithms are certainly harder than the ones with a single variable. For the euclidean algorithm you used as an example, the complexity is actually not trivial to figure out and it involves thinking about greatest-common-divisors instead of just looking at the source code for the algorithm's implementation.