Consider an operation like binary multiplication between one array with size (1*4) and matrix with size (4*8), as you know we obtain as the output an array of size (1*8).
Is the complexity of this operation O(1) ?
The sizes of the matrices are specified in your question.
Since the size of the problem never changes the complexity will be O(1).
The complexity would be O(n) when you would need to treat your input data exactly once.
For example, multiplying every number in an array of length n by 2 is of complexity O(n).
Now as far as your problem is concerned, if the size of your matrices was variable then the run time would be O(n^3), if we assume a naive implementation.
See Wikipedia for more details.
Related
Why does the following code for each statement refer to big O constant (here I use 1 for the convention)?
I mean if the array size gets bigger the time complexity may get larger right? Also the number in total will get larger and larger, won't it affect the complexity?
Pseudocode:
def find_sum(given_array)
total = 0 # refers to O(1)
for each i in given array: #O(1)
total+=i #O(1)
return total #O(1)
TL;DR: Because the Big O notation is used to quantify an algorithm, with regards of how it behaves with an increment of its input.
I mean if the array size gets bigger the time complexity may get
larger right? Also the number in total will get larger and larger,
won't it affect the complexity?
You are mistaken the time taken by the algorithm with the time-complexity.
Let us start by clarifying what is Big O notation in the current context. From (source) one can read:
Big O notation is a mathematical notation that describes the limiting
behavior of a function when the argument tends towards a particular
value or infinity. (..) In computer science, big O notation is used to classify algorithms
according to how their run time or space requirements grow as the
input size grows.
Informally, in computer-science time-complexity and space-complexity theories, one can think of the Big O notation as a categorization of algorithms with a certain worst-case scenario concerning time and space, respectively. For instance, O(n):
An algorithm is said to take linear time/space, or O(n) time/space, if its time/space complexity is O(n). Informally, this means that the running time/space increases at most linearly with the size of the input (source).
So for this code:
def find_sum(given_array)
total = 0
for each i in given array:
total+=i
return total
the complexity is O(n) because with an increment of the input the complexity grows linear and not constant. More accurately Θ(n).
IMO it is not very accurate to find out the complexity like:
def find_sum(given_array)
total = 0 # refers to O(1)
for each i in given array: #O(1)
total+=i #O(1)
return total #O(1)
Since the Big O notation represents a set of functions with a certain asymptotic upper-bound; as one can read from source:
Big O notation characterizes functions according to their growth
rates: different functions with the same growth rate may be
represented using the same O notation.
More accurate would be :
def find_sum(given_array)
total = 0 # takes c1 time
for each i in given array:
total+=i # takes c2 time
return total # takes c3 time
So the time complexity would be c1 + n * c2 + c3, which can be simplified to n. And since both the lower and upper bounds of this function are the same we can use Θ(n) instead of O(n).
Why does the following code for each statement refer to big O constant (here I use 1 for the convention)?
Not sure, ask the person who wrote it. It seems clear the overall runtime is not O(1), so if that's the conclusion they arrive at, they are wrong. If they didn't mean to say that, what they wrote is either wrong or confusing.
I mean if the array size gets bigger the time complexity may get larger right?
Yes, it might. Indeed, here, it will, since you are at least iterating over the elements in the array. More elements in the array, more iterations of the loop. Straightforward.
Also the number in total will get larger and larger, won't it affect the complexity?
This is an interesting insight and the answer depends on how you conceive of numbers being represented. If you have fixed-length numeric representations (32-bit unsigned ints, double-precision floats, etc.) then addition is a constant-time operation. If you have variable-length representations (like a big integer library, or doing the algorithm by hand) then the complexity of adding would depend on the addition method used but would necessarily increase with number size (for regular add-with-carry, an upper logarithmic bound would be possible). Indeed, with variable-length representations, your complexity should at least include some parameter related to the size (perhaps max or average) of numbers in the array; otherwise, the runtime might be dominated by adding the numbers rather than looping (e.g., an array of two 1000^1000 bit integers would spend almost all time adding rather than looping).
No answer so far address the second question:
Also the number in total will get larger and larger, won't it affect the complexity?
which is very important and usually not accounted for.
The answer is, it depends on your computational model. If the underlying machine may add insanely arbitrarily large numbers in constant time, then no, it does not affect the time complexity.
A realistic machine, however, operates on values of fixed width. Modern computers happily add 64 bit quantities. Some may only add 16 bit-wide values at a time. A Turing machine - which is a base of the whole complexity theory - works with 1 bit at a time. In any case, once our numbers outgrow the register width, we must account for the fact that addition takes time proportional to the number of bits in the addends, which in this case is log(i) (or log(total), but since total grows as i*(i-1)/2, its bit width is approximately log(i*i) = 2 log(i)).
With this in mind, annotating
total+=i # O(log i)
is more prudent. Now the complexity of the loop
for each i in given array:
total+=i # O(log(i))
is sum[1..n] log(i) = log(n!) ~ n log(n). The last equality comes from the Stirling approximation of a factorial.
There is no way, that the loop:
for each i in given array:
total+=i
will run in O(1) time. Even if the size of input n is 1, asymptotic analysis will still indicate, that it runs in O(n), and not in O(1).
Asymptotic Analysis measures the time/space complexity in relation to the input size, and it does not necessarily show the exact number of operations performed.
Point, that O(1) is constant, does not mean that it's just one (1) operation, but rather it means, that this particular block (which takes O(1)) does not change when the input changes, and therefore, it has no correlation to the input, so it has a constant complexity.
O(n), on the other hand, indicates, that the complexity depends on n, and it changes depending on how the input n changes. O(n) is a linear relation, when input size and runtime have 1 to 1 correlation.
Correctly written comments would look like this:
def find_sum(given_array)
total = 0 #this is O(1), it doesn't depend on input
for each i in given array: #this is O(n), as loop will get longer as the input size gets longer
total+=i #again, O(1)
return total #again, O(1)
When talking about complexity in general, things like O(3n) tend to be simplified to O(n) and so on. This is merely theoretical, so how does complexity work in reality? Can O(3n) also be simplified to O(n)?
For example, if a task implies that solution must be in O(n) complexity and in our code we have 2 times linear search of an array, which is O(n) + O(n). So, in reality, would that solution be considered as linear complexity or not fast enough?
Note that this question is asking about real implementations, not theoretical. I'm already aware that O(n) + O(n) is simplified to O(n)?
Bear in mind that O(f(n)) does not give you the amount of real-world time that something takes: only the rate of growth as n grows. O(n) only indicates that if n doubles, the runtime doubles as well, which lumps functions together that take one second per iteration or one millennium per iteration.
For this reason, O(n) + O(n) and O(2n) are both equivalent to O(n), which is the set of functions of linear complexity, and which should be sufficient for your purposes.
Though an algorithm that takes arbitrary-sized inputs will often want the most optimal function as represented by O(f(n)), an algorithm that grows faster (e.g. O(n²)) may still be faster in practice, especially when the data set size n is limited or fixed in practice. However, learning to reason about O(f(n)) representations can help you compose algorithms to have a predictable—optimal for your use-case—upper bound.
Yes, as long as k is a constant, you can write O(kn) = O(n).
The intuition behind is that the constant k doesn't increase with the size of the input space and at some point will be incomparably small to n, so it doesn't have much influence on the overall complexity.
Yes - as long as the number k of array searches is not affected by the input size, even for inputs that are too big to be possible in practice, O(kn) = O(n). The main idea of the O notation is to emphasize how the computation time increases with the size of the input, and so constant factors that stay the same no matter how big the input is aren't of interest.
An example of an incorrect way to apply this is to say that you can perform selection sort in linear time because you can only fit about one billion numbers in memory, and so selection sort is merely one billion array searches. However, with an ideal computer with infinite memory, your algorithm would not be able to handle more than one billion numbers, and so it is not a correct sorting algorithm (algorithms must be able to handle arbitrarily large inputs unless you specify a limit as a part of the problem statement); it is merely a correct algorithm for sorting up to one billion numbers.
(As a matter of fact, once you put a limit on the input size, most algorithms will become constant-time because for all inputs within your limit, the algorithm will solve it using at most the amount of time that is required for the biggest / most difficult input.)
I'm currently learning complexity (or efficiency however you call it), and I read about it in a book I got.
There is written something which I find pretty senseless and I need an explanation. I've tried looking online but I didn't find an answer for this certain example that they're giving.
For an algorithm that gets the max number in a single-dimensional array the size of n the input length would be n.
"For an algorithm that gets the max number in a two-dimensional array the size of n*n the input length would still be n."
I don't understand why the input length would be 'n' in both cases even though for the two-dimensional you have to go through n*n numbers...
It says
input length = the amount of work done ...
doesn't make any sense to me.
Would anyone care to explain? They certainly don't explain this there.
It's a common misconception (much seen here on SO) that the complexity of a scan across a 2D array with n*n elements is O(n^2). It's not, it's O(n). A scan is a linear operation, one element after another.
The 2D array is a polite fiction, it is really just a convenience for accessing a 1D array. After all, in languages which implement arrays properly (i.e. none of this array of pointers to blocks of memory) a 2D array is just a set of adjacent memory locations. And even in languages which do implement 2D arrays as arrays of pointers they're just linear segments of memory with interruptions
If a scan across a 2D array were O(n^2) then you could magically transform it to O(n) by ignoring the 2d-ness and just scanning the underlying 1d block of memory.
O(n^2) describes a different complexity class of operations such as those in which each pair of elements in the input is operated upon.
Reading in the comments that this book is written in Hebrew I would assume that the issue is a translation error or some other error in proofreading. The definition given in the comments of input length "input length is the measurement that indicates the work load of an algorithm" doesn't match what you would assume the term means at all in English.
To answer the question about complexity, they are reusing the variable 'n' in multiple places which makes it slightly confusing. They use 'n' to describe the dimension of the array and to describe the complexity. O(n) simply means the complexity is linear to the input. O(n^2) would be an exponential complexity. In this case with an array of n*n elements the input is n*n or n^2, but the complexity of the algorithm is still O(n) (or linear). This is because the algorithm still only operates on each input element once, whether the input is n or n*n. It would still be linear if it operated one each element 2 or three times as 3n and n are both linear functions (any x*n would be linear).
I hope this helps.
Big-O notation is used to classify TYPES of algorithms (complexity classes), not necessarily how much time it will ACTUALLY take to run. For instance O(cn) is just O(n) where c is a constant.
n is the size of the input whether that input is an nxn matrix or just an 'n' length array. The big-O 'n' and the program variable name are not referring to the same thing.
I am currently trying to learn time complexity of algorithms, big-o notation and so on. However, some point confuses me a lot. I know that most of the time, the input size of an array or whatever we are dealing with determines the running time of the algorithm. Let's say I have an unsorted array with size N and I want to find the maximum element of this array without using any special algorithm. I just want to iterate over the array and find the maximum element. Since the size of my array is N, this process runs at O(N) or linear time. Let M is an integer that is the square root of N. So N can be written as the square of M that is M*M or M^2. So, I think there is nothing wrong if I want to replace N with M^2. I know that M^2 is also the size of my array so my big-o notation could be written as O(M^2). So, my new running time looks like running in quadratic time. Why does this happen?
You are correct, if it happens to be that you have some variable M such that M^2 ~= N is always true, you could say the algorithm runs in O(M^2).
But, note that now - the algorithm runs in quadratic related to M, and not quadratic time related to the input, it is still linear related to the size of the input.
The important thing here is defining linear/quadratic, etc.
More precisely, you have to detail linear/quadratic, etc. with respect to something (N or M for your example). The most natural choice is to study the complexity wrt. the size of the input (N for your example).
Another trap for big integers is that the size of n is log(n). So for instance if you loop over all smaller integers, your algorithm is not polynomial.
I have a question, what does it mean to find the big-o order of the memory required by an algorithm?
Like what's the difference between that and the big o operations?
E.g
a question asks
Given the following pseudo-code, with an initialized two dimensional array A, with both dimensions of size n:
for i <- 1 to n do
for j <- 1 to n-i do
A[i][j]= i + j
Wouldn't the big o notation for memory just be n^2 and the computations also be n^2?
Big-Oh is about how something grows according to something else (technically the limit on how something grows). The most common introductory usage is for the something to be how fast an algorithm runs according to the size of inputs.
There is nothing that says you can't have the something be how much memory is used according to the size of the input.
In your example, since there is a bucket in the array for everything in i and j, the space requirements grow as O(i*j), which is O(n^2)
But if your algorithm was instead keeping track of the largest sum, and not the sums of every number in each array, the runtime complexity would still be O(n^2) while the space complexity would be constant, as the algorithm only ever needs to keep track of current i, current j, current max, and the max being tested.
Big-O order of memory means how does the number of bytes needed to execute the algorithm vary as the number of elements processed increases. In your example, I think the Big-O order is n squared, because the data is stored in a square array of size nxn.
The big-O order of operations means how does the number of calculations needed to execute the algorithm vary as the number of elements processed increases.
Yes you are correct the space and time complexity for the above pseudo code is n^2.
But for the below code the space or memory complexity is 1 and but time complexity is n^2.
I usually go by the assignments etc done within the code which gives you the memory complexity.
for i <- 1 to n do
for j <- 1 to n-i do
A[0][0]= i + j
I honestly never heard of "big O for memory" but I can easily guess it is only loosely relater to the computation time - probably only setting a lower bound.
As an example, it is easy to design an algorithm which uses n^2 memory and n^3 computation, but i think it is impossible to do the other way round - you cannot process n^2 data with n complexity computationally.
Your algorithm has complexity 1/2 * n^ 2, thus O(n^2)
If A is given to your algorithm, then the space complexity is O(1). Iterating over an existing 2D array and writing values to existing memory locations uses no additional memory.
However, if the algorithm allocates A, then the space complexity is O(n2).
The time complexity is O(n2) either way.