I'm looking for an intuitive, real-world example of a problem that takes (worst case) exponential time complexity to solve for a talk I am giving.
Here are examples for other time complexities I have come up with (many of them taken from this SO question):
O(1) - determining if a number is odd or even
O(log N) - finding a word in the dictionary (using binary search)
O(N) - reading a book
O(N log N) - sorting a deck of playing cards (using merge sort)
O(N^2) - checking if you have everything on your shopping list in your trolley
O(infinity) - tossing a coin until it lands on heads
Any ideas?
O(10^N): trying to break a password by testing every possible combination (assuming numerical password of length N)
p.s. why is your last example is of complexity O(infinity) ? it's linear search O(N) .. there are less than 7 billion people in the world.
A pizza restaurant has several toppings to choose from
Pepperoni
Chilli peppers
Pineapple (don't knock it until you've tried it!)
Customers may choose any combination of toppings or none at all for their pizza. Now consider an algorithm that finds every possible unique combination of toppings. This is an exponential algorithm with time complexity O(2^n).
Look how the possible combinations grow (exponentially) when you add a new topping to the menu:
0 toppings: 1 combination (no toppings at all)
1 toppings: 2 combinations (none, a)
2 toppings: 4 combinations (none, a, b, ab)
3 toppings: 8 combinations (none, a, b, c, ab, ac, bc, abc)
...
...
10 toppings: 1,024 combinations
20 toppings: 1,048,576 combinations
So with just 20 types of toppings, there are over 1 million possible combinations!
A brute-force and naive n-queens problem's solution.
You have to place n queens on a n*n board without them to be taken by others.
while there are untried configs,
go to next solution and
test it
Assuming every queen is on a given row, there are n possibilities for the queen to be placed and n for the (n-1) other queens (because duplicate rows are not checked).
Therefore, you've got a O(n^n) complexity
The brute force solution of the traveling salesman problem is O(n!) which is approximately O(N^N)
What about finding a subset of integers within a set such that their sum is a designated value X?
I believe this has complexity O(2^(n/2))
Related
i have an integer array, and i need to find a subset of this array with max 3 elements which is equal to W.
Can i solve this problem using knapsack? or i need to calculate every 1-2-3 element combination of the array?
Thanks already
To look for 3 elements that sum to W, this is exactly the 3SUM problem.
It can be solved in O(n2) time by either:
Inserting each number into a hash table, then, for each combination of two numbers a and b, checking whether W-a-b exists in the hash table.
Sorting the array, then, for each element a, looking right for two elements that sums to W-a using 2 iterators from either side.
If the integer range is in the range [-u, u], you can solve your problem using a Fast Fourier transform in O(n + u log u). See the above link for more details on either this or one of the above approaches.
Since your problem is dependent on solving 3SUM, which is a well-known problem, you're very unlikely to find a solution with better running time than the above well-known solutions for 3SUM.
To look for 1 element:
You can do a simple linear search (O(n)).
To look for 2 elements:
This can be solved by simply checking each combination of 2 elements in O(n2) (you needn't do something more complex as the asymptotic running time of 3 elements will result in O(n2) total time regardless of how efficient this is).
It can also be solved in O(n) or O(n log n) using methods identical to those described above for 3SUM.
Overall running time:
O(n2) or O(n + u log u), depending on which method was used to solve the 3SUM part.
For a given set of numbers
3 5 3 6 3 4 10 4 5 2
I wish to find all the **triplets** which form a arithmetic progression.
like (3,3,3) (3,4,5) (6,4,2) (3,4,5)
I have a trivial O(n^3) solution. I was wondering if it can be done it time O(n^2) or less.
Any help is highly appreciated.
O(n^2 * logn) can be achieved by:
Sort the array - O(nlogn)
iterate all pairs (O(n^2) of those) - and for each pair (x,y) do a binary search to see if you have: max{x,y} + abs(x-y) or min{x,y} - abs(x-y) as an element.
Special care should be taken for pairs where x==y - but it can be easily solved within the same time complexity.
Note that this solution will give you 1 occurance of each triplet (no duplicates).
(EDIT: by using a hash table (histogram if you care for the number of triplets ) and look in it instead of sorting the array and using binary search - you can reduce the time to O(n^2) on average, with the cost of O(n) additional space).
Without the 1 occurance drawback - it cannot be done better then O(n^3), because there could be O(n^3) such triplets, for example in the array [1,1,1,...,1] - you have chose(3,n) such triplets.
One can use hashing to solve it in O(n^2) by choosing a middle element and then choosing first and last element in O(n).
This is simple question of finding two numbers in a array whose sum is fixed. Here, a+c should be 2b.
Therefore, all I look for a & c such that a+c=2i.
Can you explain what makes an algoritm to be O(log n)?
I would appreciate if you can show it with a simple code.
Thanks
log(n)/log(i) is the solution to the reccurence relation
f(n) = f(n/i) + c
Each time you encouter a function which can be written as a recursive call where the size of the input is divided by a constant at each iteration.Something like
function(input, N){
do some constant work
return function(input, N/c);
}
Then you have a Theta(logn) complexity.
Examples :
When you search in a balanced search tree : take the tree, of size n, if equal to the root return (constant), or search in the left subtree if smaller (size n/2), search in the right subtree (size n/2) if bigger.
Exponentialtion a^n: if n = 1, then return a. Otherwise exponentiate a^(n/2) = b ( problem of size n/2), multiply b by b, and if n even, multiply by a once.
See Plain English explanation of Big O. It’s explained there.
The simplest definition I can give for Big-O notation is this:
Big-O notation is a relative representation of the complexity of an algorithm.
There are some important and deliberately chosen words in that
sentence:
relative: you can only compare apples to apples. You can't compare an algorithm to do arithmetic multiplication to an algorithm
that sorts a list of integers. But two algorithms that do arithmetic
operations (one multiplication, one addition) will tell you something
meaningful;
representation: Big-O (in its simplest form) reduces the comparison between algorithms to a single variable. That variable is
chosen based on observations or assumptions. For example, sorting
algorithms are typically compared based on comparison operations
(comparing two nodes to determine their relative ordering). This
assumes that comparison is expensive. But what if comparison is cheap
but swapping is expensive? It changes the comparison; and
complexity: if it takes me one second to sort 10,000 elements how long will it take me to sort one million? Complexity in this
instance is a relative measure to something else.
Come back and reread the above when you've read the rest.
The best example of Big-O I can think of is doing arithmetic. Take
two numbers (123456 and 789012). The basic arithmetic operations we
learnt in school were:
addition;
subtraction;
multiplication; and
division.
Each of these is an operation or a problem. A method of solving these
is called an algorithm.
Addition is the simplest. You line the numbers up (to the right) and
add the digits in a column writing the last number of that addition in
the result. The 'tens' part of that number is carried over to the
next column.
Let's assume that the addition of these numbers is the most expensive
operation in this algorithm. It stands to reason that to add these
two numbers together we have to add together 6 digits (and possibly
carry a 7th). If we add two 100 digit numbers together we have to do
100 additions. If we add two 10,000 digit numbers we have to do
10,000 additions.
See the pattern? The complexity (being the number of operations)
is directly proportional to the number of digits n in the larger
number. We call this O(n) or linear complexity.
Subtraction is similar (except you may need to borrow instead of
carry).
Multiplication is different. You line the numbers up, take the first
digit in the bottom number and multiply it in turn against each digit
in the top number and so on through each digit. So to multiply our
two 6 digit numbers we must do 36 multiplications. We may need to do
as many as 10 or 11 column adds to get the end result too.
If we have two 100-digit numbers we need to do 10,000 multiplications
and 200 adds. For two one million digit numbers we need to do one
trillion (1012) multiplications and two million adds.
As the algorithm scales with n-squared, this is O(n2)
or quadratic complexity. This is a good time to introduce another
important concept:
We only care about the most significant portion of complexity.
The astute may have realized that we could express the number of
operations as: n2 + 2n. But as you saw from our example
with two numbers of a million digits apiece, the second term (2n)
becomes insignificant (accounting for 0.00002% of the total operations
by that stage).
The Telephone Book
The next best example I can think of is the telephone book, normally
called the White Pages or similar but it'll vary from country to
country. But I'm talking about the one that lists people by surname
and then initials or first name, possibly address and then telephone
numbers.
Now if you were instructing a computer to look up the phone number for
"John Smith", what would you do? Ignoring the fact that you could
guess how far in the S's started (let's assume you can't), what would
you do?
A typical implementation might be to open up to the middle, take the
500,000th and compare it to "Smith". If it happens to be
"Smith, John", we just got real lucky. Far more likely is that "John
Smith" will be before or after that name. If it's after we then
divide the last half of the phone book in half and repeat. If it's
before then we divide the first half of the phone book in half and
repeat. And so on.
This is called a bisection search and is used every day in
programming whether you realize it or not.
So if you want to find a name in a phone book of a million names you
can actually find any name by doing this at most 21 or so times (I
might be off by 1). In comparing search algorithms we decide that
this comparison is our 'n'.
For a phone book of 3 names it takes 2 comparisons (at most). For
7 it takes at most 3. For 15 it takes 4. ... For
1,000,000 it takes 21 or so.
That is staggeringly good isn't it?
In Big-O terms this is O(log n) or logarithmic complexity.
Now the logarithm in question could be ln (base e), log10,
log2 or some other base. It doesn't matter it's still
O(log n) just like O(2n2) and O(100n2) are still
both O(n2).
Wikipedia gives some examples of orders of common functions:
O(log(n)), Finding an item in a sorted array with a binary search or a balanced search tree as well as all operations in a Binomial heap.
The most simple example: binary search
Principle: In every step prunning half of the possible search space (searched array), after at most log(n) steps the algorithm must terminate (because of the cutting in half).
In T(n) = 2T(n/2) + M(n), where does the 2 in front of T come from. n/2 because it is dividing, and M(n) is linear, but I can't figure out what the 2 is for?
2, because you are performing the operation on the two subsets. See the master theorem.
The recurrence relation is similar to what you get in Merge Sort. The time complexity would be O(n log n)
This says that the time cost of the problem of size n comes from dividing the problem in half (i.e., T(n/2)) and solving it for both halves (2 T(n/2)) plus some fix-up cost (i.e., M(n)).
So, the 2 is because you divide the problem in half and solve both halves.
The 2 represents how many times you're going to call the recurring function.
For example, if you had a tree that had 4 children, you would expect a 4 for that value. In this case, you're recurring twice.
I was just answering a question about different approaches for picking the partition in a quicksort implementation and came up with a question that I honestly don't know how to answer. It's a bit math-heavy, and this may be the wrong site on which to ask this, so if this needs to move please let me know and I'll gladly migrate it elsewhere.
It's well-known that a quicksort implementation that picks its pivots uniformly at random will end up running in expected O(n lg n) time (there's a nice proof of this on Wikipedia). However, due to the cost of generating random numbers, many quicksort implementations don't pick pivots randomly, but instead rely on a "median-of-three" approach in which three elements are chosen deterministically and of which the median is chosen as the pivot. This is known to degenerate to O(n2) in the worst-case (see this great paper on how to generate those worst-case inputs, for example).
Now, suppose that we combine these two approaches by picking three random elements from the sequence and using their median as the choice of pivot. I know that this also guarantees O(n lg n) average-case runtime using a slightly different proof than the one for the regular randomized quicksort. However, I have no idea what the constant factor in front of the n lg n term is in this particular quicksort implementation. For regular randomized quicksort Wikipedia lists the actual runtime of randomized quicksort as requiring at most 1.39 n lg n comparisons (using lg as the binary logarithm).
My question is this: does anyone know of a way to derive the constant factor for the number of comparisons made using a "median-of-three" randomized quicksort? If we go even more generally, is there an expression for the constant factor on quicksort using a randomized median-of-k approach? I'm curious because I think it would be fascinating to see if there is some "sweet spot" of this approach that makes fewer comparisons than other randomized quicksort implementations. I mean, wouldn't it be cool to be able to say that randomized quicksort with a randomized median-of-six pivot choice makes the fewest comparisons? Or be able to conclusively say that you should just pick a pivot element at random?
Here's a heuristic derivation of the constant. I think it can be made rigorous, with a lot more effort.
Let P be a continuous random variable with values in [0, 1]. Intuitively, P is the fraction of values less than the pivot. We're looking to find the constant c such that
c n lg n = E[n + c P n lg (P n) + c (1 - P) n lg ((1 - P) n)].
A little bit of algebra later, we have
c = 1/E[-P lg P - (1 - P) lg (1 - P))].
In other words, c is the reciprocal of the expected entropy of the Bernoulli distribution with mean P. Intuitively, for each element, we need to compare it to pivots in a way that yields about lg n bits of information.
When P is uniform, the pdf of P is 1. The constant is
In[1]:= -1/NIntegrate[x Log[2, x] + (1 - x) Log[2, 1 - x], {x, 0, 1}]
Out[1]= 1.38629
When the pivot is a median of 3, the pdf of P is 6 x (1 - x). The constant is
In[2]:= -1/NIntegrate[6 x (1 - x) (x Log[2, x] + (1 - x) Log[2, 1 - x]), {x, 0, 1}]
Out[2]= 1.18825
The constant for the usual randomized quicksort is easy to compute because the probability that two elements k locations apart are compared is exactly 2/(k+1): the probability that one of the those two elements is chosen as a pivot before any of the k-1 elements between them. Unfortunately nothing so clever applies to your algorithm.
I'm hesitant to attempt your bolded question because I can answer your "underlying" question: asymptotically speaking, there is no "sweet spot". The total added cost of computing medians of k elements, even O(n1 - ε) elements, is linear, and the constant for the n log n term decreases with the array being split more evenly. The catch is of course constants on the linear term that are spectacularly impractical, highlighting one of the drawbacks of asymptotic analysis.
Based on my comments below, I guess k = O(nα) for 0 < α < 1 is the "sweet spot".
If the initial state of the set is randomly ordered, you will get the exact same constant factor for randomly picking three items to calculate the median as when picking three items deterministically.
The motive for picking item by random would be that the deterministic method would give a result that is worse than the average. If the deterministic method gives a good median, you can't improve on it by picking items by random.
So, which ever method gives the best result depends on the input data, it can't be determined for every possible set.
The only sure way to lower the constant factor is to increase the number of items that you use to calcuate the median, but at some point calculating the median will be more expensive than what you gain by getting a better median value.
Yes, it does. Bentley and McIlroy, authors of the C standard library's qsort function, wrote in their paper, Engineering a Sort Function the following numbers:
1.386 n lg n average comparisons using first, middle or a randomized pivot
1.188 n lg n average comparisons using a median of 3 pivot
1.094 n lg n average comparisons using a median of 3 medians pivot
According to the above paper:
Our final code therefore chooses the middle element of smaller arrays,
the median of the first, middle and last elements of a mid-sized
array, and the pseudo-median of nine evenly spaced elements of a large
array.
Just a thought: If you use the median-of-three approach, and you find it to be better, why not use a median-of-five or median-of-eleven approach? And while you are on it, maybe one can think of a median-of-n optimization... hmmm... Ok, that is obviously a bad idea (since you would have to sort your sequence for that...).
Basically, to choose your pivot element as the median-of-m elements, you sort those m elements, right? Therefore, I'd simply guess, that one of the constants you are looking for is "2": By first sorting 3 elements to choose your pivot you execute how many additional comparisons? Lets say its 2. You do this inside the quicksort over and over again. A basic conclusion would be that the median-of-3 is therefore 2-times slower then the simple random quicksort.
But what is working for you here? That you get a better device-and-conquer-distribution, and you are better protected against the degenerated case (a bit).
So, back to my infamous question at the beginning: Why not choose the pivot element from a median-of-m, m being 5, 7, n/3, or so. There must be a sweet spot where the sorting of the m elements is worse then the gain from the better divide-and-conquer behavior and quicksort. I guess, this sweet-spot is there very early -- you have to fight first against the constant factor of 2 comparisons if you choose median-of-3. It is worth an experiment, I admit, but I would not be too expectant of the result :-) But if I am wrong, and the gain is huge: don't stop at 3!