I was going through this text from Cracking the Coding Interview and something doesn't look clear to me:
Arrays and Strings
String joinWords(String[] words) {
String sentence = "";
for (String w : words) {
sentence = sentence + w;
}
return sentence;
}
On each concatenation, a new copy of the string is created, and the two strings are copied over, character by character. The first iteration requires us to copy 𝑥 characters. The second iteration requires copying 2𝑥 characters. The third iteration requires 3𝑥, and so on. The total time therefore is 𝒪(𝑥 + 2𝑥 + ... + 𝑛𝑥). This reduces to 𝒪(𝑥𝑛²).
Why is it 𝒪(𝑥𝑛²)? Because 1 + 2 + ... + n equals 𝑛(𝑛+1)/2, or 𝒪(𝑛²).
How does 𝒪(𝑥 + 2𝑥 + 𝑛𝑥) reduce to 𝒪(𝑥𝑛²)?
My analogy, assuming 𝑥 is a constant 1, then 𝑛 is 2(𝑥 + 2𝑥) == 3
From the book (𝑥2²) == 4 assuming 𝑥 is a constant 1
Is the algorithm analysis in the above code correct?
In the above calculation O(x + 2x + ... + nx)
x + 2x + ... + nx is expanded as x(n(n+1)/2)
which is x((n^2+n/2)) since we neglect constants and in time complexity calculation and take the value with the largest power value it is taken as 𝑂(𝑛2).
It is similar to taking 𝑂(3𝑛) as 𝑂(𝑛).
To rationalize how asymptotic notations ignore constant factors, I usually think of it like this: asymptotic complexity isn't for comparing the performance of different algorithms, it's for understanding how the performance of individual algorithms scales with respect to the input size.
For instance, we say that a function that takes 3𝑛 steps is 𝑂(𝑛), because, roughly speaking, for large enough inputs, doubling the input size will no more than double the number of steps taken. Similarly, 𝑂(𝑛2) means that doubling the input size will at most quadruple the number of steps, and 𝑂(log𝑛) means that doubling the input size will increase the number of steps by at most some constant.
It's a tool for saying which algorithms scale better, not which ones are absolutely faster.
For more info : https://www.quora.com/Why-do-we-leave-the-constants-while-calculating-time-complexity-for-algorithms
Yes, the book's analysis is roughly correct, although it silently assumes all words have the same length x.
You are right that 2(𝑥+2𝑥) == 3, while the book's formula gives 𝑥2² == 4, but in big-𝒪 notation we don't look at the exact value, but the order of magnitude.
There are ½𝑥𝑛(𝑛+1) characters copied. This is because the expression 1+2+...+𝑛 is a triangle number, equal to ½𝑛(𝑛+1). Only remains to multiply with 𝑥.
For 𝑥 == 1 and 𝑛 == 2 it gives your result, i.e. 3
We can write this as ½𝑥𝑛² + ½𝑥𝑛. Now, when we go to big-𝒪 notation, only the most significant term needs to be retained, and any constant coefficient can be dropped, and so:
𝒪[½𝑥𝑛² + ½𝑥𝑛] == 𝒪[½𝑥𝑛²] == 𝒪(𝑛²)
Obviously this means that the expression will give a different value for a given 𝑥 and 𝑛, but in big-𝒪 notation that is not what is the point. It gives an order of magnitude.
Related
I am learning about calculating the time complexity of an algorithm, and there are two examples that I can't get my head around why their time complexity is different than I calculated.
After doing the reading I learned that the for-loop with counter increasing once each iteration has the time complexity of O(n) and the nested for-loop with different iteration conditions is O(n*m).
This is the first question where I provided the time complexity to be O(n) but the solution says it was O(1):
function PrintColours():
colours = { "Red", "Green", "Blue", "Grey" }
foreach colour in colours:
print(colour)
This is the second one where I provided the time complexity to be O(n^2) but the solution says its O(n):
function CalculateAverageFromTable(values, total_rows, total_columns):
sum = 0
n = 0
for y from 0 to total_rows:
for x from 0 to total_columns:
sum += values[y][x]
n += 1
return sum / n
What am I getting wrong with these two questions?
There are several ways for denoting the runtime of an algorithm. One of most used notation is the Big - O notation.
Link to Wikipedia: https://en.wikipedia.org/wiki/Big_O_notation
big O notation is used to classify algorithms according to how their
run time or space requirements grow as the input size grows.
Now, while the mathematical definition of the notation might be daunting, you can think of it as a polynomial function of input size where you strip away all the constants and lower degree polynomials.
For ex: ax^2 + bx + c in Big-O would be O(x^2) (we stripped away all the constants a,b and c and lower degree polynomial bx)
Now, let's consider your examples. But before doing so, let's assume each operation takes a constant time c.
First example:
Input is: colours = { "Red", "Green", "Blue", "Grey" } and you are looping through these elements in your for loop. As the input size is four, the runtime would be 4 * c. It's constant runtime and constant runtime is written as O(1) in Big-O
Second example:
The inner for loop runs for total_columns times and it has two operations
for x from 0 to total_columns:
sum += values[y][x]
n += 1
So, it'll take 2c * total_columns times. And, the outer for loop runs for total_rows times, resulting in total time of total_rows * (2c * total_columns) = 2c * total_rows * total_columns. In Big-O it'd be written as O(total_rows * total_columns) (we stripped away the constant)
When you get out of outer loop, n which was set to 0 initially, would become total_rows * total_columns and that's why they mentioned the answer to be O(n).
One good definition of time complexity is:
"It is the number of operations an algorithm performs to complete its
task with respect to the input size".
If we think the following question input size can be defined as X= total_rows*total_columns. Then, what is the number of operations? It is X again because there will be X addition because of the operation sum += values[y][x] (neglect increment operation for n += 1 for simplicity). Then, think that we double array size from X to 2*X. How many operations there will be? It is 2*X again. As you can see, increase in number of operations is linear when we increase input size which makes time complexity O(N).
function CalculateAverageFromTable(values, total_rows, total_columns):
sum = 0
n = 0
for y from 0 to total_rows:
for x from 0 to total_columns:
sum += values[y][x]
n += 1
return sum / n
For your first question, the reason is that colours is a set. In python, {} defines a set. Accessing elements from unordered set is O(1) time complexity regardless of the input size. For furher information you can check here.
When we have a recursive function to generate parentheses with N valid parentheses, the time complexity is that of the Catalan number. This doesn't make sense to me.
My analysis of the time complexity is that, we have two operations at every node of the recursion tree. We can either add a close bracket or an opening bracket. So we make two recursive calls.
T(n) = 2 * T(N - 1) = O(2^N)
I get O(2^N) as my time complexity -- not the Catalan number. The Catalan number is so arbitrary to me -- it doesn't make sense. Could anyone explain it a bit further?
In your assumption, you explore all cases that can be formed by the characters '(' and ')'. However, it is possible to eliminate some of those cases, isn't it? For instance, we know that for an input N = 4, "))((" is not a valid/balanced string. In fact, we know this to be true from the moment we put the first character of that string. Here's a recursive implementation in Python, just so that we can observe it through an example.
def generate(index, N, s, depth):
if index == N:
print s
if depth > 0:
generate(index + 1, N, s + ')', depth - 1)
if depth < N:
generate(index + 1, N, s + '(', depth + 1)
Essentially, in a recursive implementation, you keep a score of the current depth. Whenever that score is less than 0, you know that your string becomes unbalanced, thus there is no point in exploring further. So, contrary to what you assumed, you do not explore both the subproblems.
If you think about it, the problem is simply finding the number of valid permutations of N = 2 * K different characters. At the first(leftmost) position, you can place K characters. (i.e. all the '(') In the second position, you can either place one of the ')' characters, or you can place one of the remaining K-1 '(' characters. With this approach, using permutation with repetition, you can find that the complexity of the problem you mentioned is, indeed, equivalent to the Kth Catalan number.
Basically, for a string of length 2N, you have two different characters of which you have N, each. Using permutation with repetition, all the possible permutations for this string would be (2N)! / (N! N!). Well, the formula for the Nth Catalan number is just that value, divided by an additional (N+1), as you can see in the relevant Wikipedia article. If you consider the cases where you do not handle the unbalanced strings I mentioned above, you can see that (N+1) factor is due to the cases where you don't compute both the subproblems.
This question is about whether there is some abstract similarity between the solutions that leads to the appearance of log in problems such as sorting and searching. Or, more simply, why does log appear so frequently in algorithmic complexity?
Logarithms often show up when a problem can be repeatedly reduced in size by a multiplicative factor. By definition there's a logarithmic number of steps required to reduce the problem to constant size (e.g. size 1).
A typical example would be repeatedly eliminating half of the dataset, as is done in binary search. This gives O(log2(n)) complexity. Some sorting algorithms work by repeatedly splitting the dataset in half, and therefore also have a logarithmic term in their time complexity.
More generally, logarithms frequently show up in solutions to divide-and-conquer recurrence relations. See Master Theorem in Wikipedia for further discussion.
log appears a lot in algorithm complexity especially in recursive algorithms..
lets take at a binary search for example.
you have a sorted array A of 100 elements and your looking for the number 15..
in a binary search you will look at the middle element (50) and compare it to 15.. if element is greater than 15 then you find the middle element between 50 and 100 which is 75.. and compare again.. if 15 is greater than the element at 75 then you look at the element between 75 and 100 which is element 87... you continue to do this until you find the element or until there is no more middle number...
each time you do this method of checking the middle number you cut the total number of elements remaining to search in half..
so the first pass will give you O(n/2) complexity.. the next pass will be O(n/4)... O(n/8) and so on..
to represent this pattern we use logs..
since we are cutting the number of elements to search in half with each pass of the algorithm that becomes the log base so the binary search will yield a O(log2(n)) complexity
most algorithms try to 'cut' the number of operations down to as few as possible by breaking the original data into separate parts to solve and that is why log shows up so often
log appears very often in computer science because of the boolean logic. Everything can be reduced to true vs false or 1 vs 0 or to be or not to be. If you have an if statement you have one option, otherwise you have the other option. This can be applied for bits (you have 0 or 1) or in high impact problems, but there is a decision. And as it is in the real life, when you take a decision, you don't care about the problems that could have happened if you had decided otherwise. This is why log2(n) appears very often.
Then, every situation that is more complicated ( E.g.: choose one possible state from 3 states ) can be reduced to log2(n) => the logarithm base doesn't matter (a constant doesn't influence trend for a function - it has the same degree ):
Mathematical proof:
loga(y) 1
logx(y) = ------- = -------- * loga(y) = constant * loga(y)
loga(x) loga(x)
Proof for programmers:
switch(a) {
case 1: ... ;
case 2: ... ;
...
default: ... ;
}
is similar to:
if (a == 1) {
...
} else {
if ( a == 2 ) {
...
}
...
}
( a switch for k options is equivalent to k-1 if-else statements, where k = constant )
But why log ? Because it is the inverse for an exponential. At the first decision you break the big problem into 2 parts. Then you break only the "good" half in 2 parts, etc.
n = n/2^0 // step 1
n/2 = n/2^1 // step 2
n/4 = n/2^2 // step 3
n/8 = n/2^3 // step 4
...
n/2^i = n/2^i // step i+1
Q: How many steps are there ?
A: i+1 ( from 0 to i )
Because it stops when you find the wanted element (there are no other decisions you can take) => n = 2^i. If we apply the logarithm, base 2:
log2(n) = log2(2^i)
log2(n) = i
=> i + 1 = log2(n) + 1
But a constant doesn't influence the complexity => you have ~log2(n) steps.
I've been doing some reading on time complexity, and I've got the basics covered. To reinforce the concept, I took a look at an answer I recently gave here on SO. The question is now closed, for reason, but that's not the point. I can't figure out what the complexity of my answer is, and the question was closed before I could get any useful feedback.
The task was to find the first unique character in a string. My answer was something simple:
public String firstLonelyChar(String input)
{
while(input.length() > 0)
{
int curLength = input.length();
String first = String.valueOf(input.charAt(0));
input = input.replaceAll(first, "");
if(input.length() == curLength - 1)
return first;
}
return null;
}
Link to an ideone example
My first thought was that since it looks at each character, then looks at each again during replaceAll(), it would be O(n^2).
However, it got me to thinking about the way it actually works. For each character examined, it then removes all instances of that character in the string. So n is constantly shrinking. How does that factor into it? Does that make it O(log n), or is there something I'm not seeing?
What I'm asking:
What is the time complexity of the algorithm as written, and why?
What I'm not asking:
I'm not looking for suggestions to improve it. I get that there are probably better ways to do this. I'm trying to understand the concept of time complexity better, not find the best solution.
The worst time complexity you will have is for the string aabb... and so on each character repeated exactly twice. Now this depends on the size of your alphabet let's say that is S. Let's also annotate the length of your initial string with L. So for each letter you will have to iterate over the whole String. However first time you do that the String will be of size L, second time L-2 and so on. Overall you will have to perform in the order of L + (L-2) + ... + (L- S*2) operations and that is L*S - 2*S*(S+1), assuming L is more than 2*S.
By the way if the size of your alphabet is constant, and I suppose it is, the complexity of your code is O(L)(though with a big constant).
The worst case is O(n^2) where n is the length of the input string. Imagine the case where every character is doubled except the last one, like "aabbccddeeffg". Then there are n/2 loop iterations, and each call to replaceAll has to scan the entire remaining string, which is also proportional to n.
Edit: As Ivaylo points out, if the size of your alphabet is constant, it's technically O(n) since you never consider any character more than once.
Let's mark:
m = number of unique letters in the word
n = input length
This is the complexity calculation:
The main loop goes at most m times, because there are m different letters,
the .Replaceall checks at most O(n) comparisons in each cycle.
the total is: O(m*n)
an example for O(m*n) cycle is: input = aabbccdd,
m=4, n=8
the algorithm stages:
1. input = aabbccdd, complex - 8
2. input = bbccdd, complex = 6
3. input = ccdd, complex = 4
4. input = dd, complex = 2
total = 8+6+4+2 = 20 = O(m*n)
Let m be the size of your alphabet, and let n be the length of your string. The worse case would be to uniformly distribute your string's characters between the alphabet letters, meaning you'll have n / m characters for each letter in your alphabet, let's mark this quantity with q. For example, the string aabbccddeeffgghh is the uniformly distribution of 16 characters between the letters a-h, so here n=16 and m=8 and you have q=2 characters for each letter.
Now, your algorithm is actually going over the letters of the alphabet (it just uses the order which they appear in the string), and for each iteration it has to go over the length of the string (n) and shrink it by q (n -= q). So over all the number of operation you do in the worst case are:
s = n + n-(1*q) + ... + n-((m-1)*q)
You can see that s is the sum of the first m elements of the arithmetic series:
s = (n + n-((m-1)*q) * m / 2 =
(n + n-((m-1)*(n/m)) * m / 2 ~ n * m / 2
Question 01:
How can I find T (1) when I measure the complexity of an algorithm?
For example
I have this algorithm
Int Max1 (int *X, int N)
{
int a ;
if (N==1) return X[0] ;
a = Max1 (X, N‐1);
if (a > X[N‐1]) return a;
else return X[N‐1];
}
How can I find T(1)?
Question 2 :
T(n)= T(n-1) + 1 ==> O(n)
what is the meaning of the "1" in this equation
cordially
Max1(X,N-1) Is the actual algorithm the rest is a few checks which would be O(1)
as regardless of input the time taken will be the same.
The Max1 function I can only assume is finding array highest number in array this would be O(n) as it will increase in time in a linear fashion to the number of input n.
Also as far as I can tell 1 stands for 1 in most algorithms only letters have variable meanings, if you mean how they got
T(n-1) + 1 to O(n), it is due to the fact you ignore coefficients and lower order terms so the 1 is both cases is ignored to make O(n)
Answer 1. You are looking for a complexity. You must decide what case complexity you want: best, worst, or average. Depending on what you pick, you find T(1) in different ways:
Best: Think of the easiest input of length 1 that your algorithm could get. If you're searching for an element in a list, the best case is that the element is the first thing in the list, and you can have T(1) = 1.
Worst: Think of the hardest input of length 1 that your algorithm could get. Maybe your linear search algorithm executes 1 instruction for most inputs of length 1, but for the list [77], you take 100 steps (this example is a bit contrived, but it's entirely possible for an algorithm to take more or less steps depending on properties of the input unrelated to the input's "size"). In this case, your T(1) = 100.
Average: Think of all the inputs of length 1 that your algorithm could get. Assign probabilities to these inputs. Then, calculate the average T(1) of all possibilities to get the average-case T(1).
In your case, for inputs of length 1, you always return, so your T(n) = O(1) (the actual number depends on how you count instructions).
Answer 2. The "1" in this context indicates a precise number of instructions, in some system of instruction counting. It is distinguished from O(1) in that O(1) could mean any number (or numbers) that do not depend on (change according to, trend with, etc.) the input. Your equation says "The time it takes to evaluate the function on an input of size n is equal to the time it takes to evaluate the function on an input of size n - 1, plus exactly one additional instruction".
T(n) is what's called a "function of n," which is to say, n is a "variable" (meaning that you can substitute in different values in its place), and each particular (valid) value of n will determine the corresponding value of T(n). Thus T(1) simply means "the value of T(n) when n is 1."
So the question is, what is the running-time of the algorithm for an input value of 1?