I am currently reading the Introduction to Algorithms book and I have a question in regard to analyzing an algorithm:
The computational cost for merge sort is c lg n according to the book and it says that
We restrict c to be a constant so that the word size does not grow arbirarily (If the word size could grow arbitrarily, we could store huge amounts of data in one word and operate on it all in constant time)
I do not understand the meaning of "constant" here. Could anyone explain clearly what this means?
Computational complexity in the study of algorithms deals with finding function(s) which provide upper and lower bounds for how much time (or space) the algorithm requires. Recall basic algebra in high school where you learned about the general point-slope formula for a line? That formula, y = mx + b, provided two parameters, m (slope), and b (y intercept), which described a line completely. Those constants (m,b) described where the line lay, and a larger slope meant that the line was steeper.
Algorithmic complexity is just a way to describe the upper (and possibly lower) bounds for how long an algorithm takes to run (and/or how much space is required). With big-O (and big-Theta) notation, you are finding a function which provides upper (and lower) bounds for the algorithm costs. The constants are just shifting the curve, not changing the shape of the curve.
We restrict c to be a constant so that the word size does not grow arbirarily (If the word size could grow arbitrarily, we could store huge amounts of data in one word and operate on it all in constant time)
On a physical computer, there is some maximum size to a machine word. On a 32-bit system, that would be 32 bits, and on a 64-bit system, it's probably 64 bits. Operations on machine words are (usually) assumed to take time O(1) even though they operate on lots of bits at the same time. For example, if you use a bitwise OR or bitwise AND on a machine word, you can think of it as performing 32 or 64 parallel OR or AND operations in a single unit of time.
When trying to build a theoretical model for a computing system, it's necessary to assume an upper bound on the maximum size of a machine word. If you don't do this, then you could claim that you could perform operations like "compute the OR of n values in time O(1)" or "add together two arbitrary-precision numbers in time O(1)," operations that you can't actually do on a real computer. Therefore, there's usually an assumption that the machine word has some maximum size so that if you do want to compute the OR of n values, you can still do so, but you can't do it instantaneously by packing all the values into one machine word and performing a single assembly instruction to get the result.
Hope this helps!
Related
In calculating the efficiency of algorithms, I have read that the exponentiation operation is not considered to be an atomic operation (like multiplication).
Is it because exponentiation is the same as the multiplication operation repeated several times over?
In principle, you can pick any set of "core" operations on numbers that you consider to take a single time unit to evaluate. However, there are a couple of reasons, though, why we typically don't count exponentiation as one of them.
Perhaps the biggest has to do with how large of an output you produce. Suppose you have two numbers x and y that are each d digits long. Then their sum x + y has (at most) d + 1 digits - barely bigger than what we started with. Their product xy has at most 2d digits - larger than what we started with, but not by a huge amount. On the other hand, the value xy has roughly yd digits, which can be significantly bigger than what we started with. (A good example of this: think about computing 100100, which has about 200 digits!) This means that simply writing down the result of the exponentiation would require a decent amount of time to complete.
This isn't to say that you couldn't consider exponentiation to be a constant-time operation. Rather, I've just never seen it done.
(Fun fact: some theory papers don't consider multiplication to be a constant-time operation, since the complexity of a hardware circuit to multiply two b-bit numbers grows quadratically with the size of b. And some theory papers don't consider addition to be constant-time either, especially when working with variable-length numbers! It's all about context. If you're dealing with "smallish" numbers that fit into machine words, then we can easily count addition and multiplication as taking constant time. If you have huge numbers - say, large primes for RSA encryption - then the size of the numbers starts to impact the algorithm's runtime and implementation.)
This is a matter of definition. For example in hardware-design and biginteger-processing multiplication is not considered an atomic operation (see e.g. this analysis of the karatsuba-algorithm).
On the level that is relevant for general purpose software-design on the other hand, multiplication can be considered as a fairly fast operation on fixed-digit numbers implemented in hardware. Exponentiation on the other hand is rarely implemented in hardware and an upper bound for the complexity can only be given in terms of the exponent, rather than the number of digits.
If an algorithm requires an integer which can contain the number n (e.g. counting the size of an input array), that integer must take on the order of log(n) space (right?).
If this is the only space which scales with n, is the space complexity of the algorithm O(logn)?
Formally, it depends on the model of computation you're using. In the classical random access machine model, with a fixed word size, simply storing the length n of the input indeed requires O(log n) space (and simple arithmetic on such numbers takes O(log n) time). On the other hand, in the transdichotomous model, where the word size is assumed to grow logarithmically with the input size n, it only requires O(1) space (and time). Other models may yield yet other answers.
In practice, most analysis of algorithms assumes that simple arithmetic on moderate-size integers (i.e. proportional to the input length) can be done in constant time and space. A practical reason for this, besides the fact that it simplifies analysis, is that in practice the logarithm of the input length cannot grow very large — even a computer capable of counting up from zero to, say, 2256, much less of reading that many bits of input, is probably forever beyond the means of humankind to build using known physics. Thus, for any conceivable realistic inputs, you can simply assume that a machine with a 256-bit word size can store the length of the input in a single word (and that machines with a smaller word size still need only a small constant number of words).
Here n is bounded i.e. n will be 32 bit signed integer as array size has certain limitation. So log(32) is bounded and its O(1)
I have the following problem
Under what circumstances can multiplication be regarded as a unit time
operation?
But I thought multiplication is always considered to be taking unit time. Was I wrong?
It depends on what N is. If N is the number of bits in an arbitrarily large number, then as the number of bits increases, it takes longer to compute the product. However, in most programming languages and applications, the size of numbers are capped to some reasonable number of bits (usually 32 or 64). In hardware, these numbers are multiplied in one step that does not depend on the size of the number.
When the number of bits is a fixed number, like 32, then it doesn't make sense to talk about asymptotic complexity, and you can treat multiplication like an O(1) operation in terms of whatever algorithm you're looking at. When can become arbitrarily large, like with Java's BigInteger class, then multiplication depends on the size of those numbers, as does the memory required to store them.
Only in those cases where you're performing operations on two numbers,of numeric type (emphasis here ,not going into the binary detail), you simply need to assume that the operation being performed is of constant time only.
It's not defined as unit time,but,more strictly, a constant time interval which doesn't change even if we increase the size of number,but, in reality the calculation does utilise subtle more time to perform calculation on large numbers). These are generally considered trivial,unless the numbers being multiplied are too large,like BigIntegers in Java,etc.
But,as soon as we move towards performing multiplication of binary strings, our complexity increases and naive method yields a complexity of O(n^2).
So,to aimplify, we perform a divide and conquer-based multiplication, also known as Karatsuba's algorithm for multiplication, which has a complexity of O(n^1.59) which reduces the total number of multiplications and additions to some lesser number of multiplications and some of the additions.
I hope I haven't misjudged the question. If so,please alert me so that
I can remove this answer. If I understood the question properly, then the other answer posted here seems
incomplete.
The expression unit time is a little ambiguous (and AFAIK not much used).
True unit time is achieved when the multiply is performed in a single clock cycle. This rarely occurs on modern processors.
If the execution time of the multiply does not depend on the particular values of the operands, we can say that it is performed in constant time.
When the operand length is bounded, so that the time never exceeds a given duration, we also say that an operation is performed in constant time.
This constant duration can be used as the timing unit of the running time, so that you count in "multiplies" instead of seconds (ops, flops).
Lastly, you can evaluate the performance of an algorithm in terms of the number of multiplies it performs, independently of the time they take.
This is a homework problem.So I am looking for hints rather than the solution. Consider a set of n numbers. Each number is 'k' digits long. Suppose 'k' is much much larger and does not fit into a single word of memory. In such a scenario what is the complexity of mergesort and radix sort?
My analysis is - Asymptotic complexity doesn't depend on the underlying architectural details like the number of words a number occupies etc. May be the constant factor changes and algorithms run slower, but the overall complexity remains the same. For instance,in languages like Python that handle arbitrarily long integers, the algorithms remain the same. But a few of my friends argue that as the number of words occupied by a number 'w' grows towards infinity, the complexity does change.
Am I on the right path?
The runtime of an algorithm can indeed depend on the number of machine words making up the input. As an example, take integer multiplication. The computer can compute the product of two one-word numbers in time O(1), but it can't compute the product of two arbitrarily-sized numbers in time O(1) because the machine has to load each word into memory as part of its computation.
As a hint for radix sort versus mergesort - the mergesort algorithm makes O(n log n) comparisons between elements, but those comparisons might not take time O(1) each. How much time does it take to compare two numbers that require k machine words each? Similarly, radix sort's runtime depends on the number of digits in the number. How many rounds of radix sort do you need if you have k machine words in each number?
Hope this helps!
You're sort of correct. This is a large part of why most complexity analysis will (somewhere, at least implicitly) state that it's working with the count of some basic operations, not actual time. You generally take for granted that most (if not all) of those basic operations (e.g., comparison, swapping, math like addition or subtraction, etc.) are constant time, which lets you translate almost directly from operation count (the actual complexity) to time consumed.
To be entirely accurate, however, asymptotic complexity is (should) normally specified in terms of a count of fundamental operations, though, not actual time consumed.
What is the space complexity of the following algorithm?
Given an array of n 32-bit signed integers, where each value is positive and less than two to the power of 30, negate all values in the array, then negate all values in the array a second time.
The question arose for me out of a discussion in the comment section here: Rearrange an array so that arr[i] becomes arr[arr[i]] with O(1) extra space
I am specifically interested in different opinions and definitions. I think subtle distinctions and definitions may be missing sometimes in some stackoverflow discussions on this subject.
Space complexity usually refers to the added space requirements for an algorithm, over and above the original data set itself. In this case, the original data set is the n 32-bit signed integers, so you're only concerned with extra storage above that.
In that case, that extra storage is is basically nothing, which translates to constant O(1) space complexity.
If you were required to create a separate array (negated, then negated again), it would be O(n) since the space required is in proportion to the original data set.
But, since you're doing the negations in-place, that's not the case.
You are confusing two different, though related, things: computer-theoretic space complexity of an algorithm, and practical memory requirements of a program.
Algorithms in computer science are normally not formulated in terms of integers of certain predefined size which is imposed by currently predominant computer architectures. If anything, they are parameterized by integer size. So "given an array of n 32-bit signed integers" should be replaced with "given an array of n k-bit signed integers".
Now if all integers of the input array are actually m<k bit wide, and all integers of the output array are also known to be m<k bit wide, and nothing else outside your algorithm imposes k bit wide integers, then sneaking k in the problem description is just cheating in order to make your complexity look better then it actually is. Likewise, saying "signed" if both input and output data is supposed to be positive is cheating.
Real-life programs don't have complexity, they have memory requirements. It is perfectly fine to say that your program does not use any extra memory if it only temporarily uses otherwise unused sign bits of your array elements. Just don't act surprised when one fine day you discover you have too large an array and you must pack it, so that it no longer has any unused bits. That is, you are reusing your algorithm in a different program with a different data representation, one that does not have any spare bits. Then you are forced to recall that the added space complexity of your algorithm is actually O(n).
Since you're interested in space complexity, the only relevant part of the question is:
"an array of n 32-bit signed integers"
From the above, the answer is pretty straightforward - O(n)
This whole blurb:
negate all values in the array, then negate all values in the array a
second time
only affects the time complexity, which seems like a poorly crafted distraction in a homework assignment.