Algorithmic Reduction (Median of medians, quicksort) - algorithm

I'm trying to better understand reduction, and I'm currently looking at the two algorithms, "Median of medians" and Quicksort.
I understand that both algorithms use a similar (effectively identical) partition subroutine to help solve their problems, which ends up making them quite similar.
Select(A[1...n],k): // Pseudocode for median of medians
m = [n/5]
for i from 1 to m:
B[i] = Select(A[5i-4..5i],3)
mom = Select(B[1..m],m/2)
r = partition(A[1..n],mom) // THIS IS THE SUBROUTINE
if k < r:
return Select(A[1..r-1],k)
else if k > r:
return Select(A[r+1..n],k-r)
else
return mom
So does the term "reduction" make any sense in regards to these two algorithms? Do any of the following make sense?
Median of Medians/Quicksort can be reduced to a partition subroutine
Median of medians reduces to quicksort
Quicksort reduces to median of medians

This really depends on your definition of "reduction."
The standard type of reduction that's usually discussed is a mapping reduction (also called a many-one reduction). A mapping reduction from problem X to problem Y is the following:
Given an input IX to problem X, transform it into an input IY to problem Y. Then, run a solver for problem Y on IY and output that answer.
In a mapping reduction, you get to make exactly one call to a subroutine that solves problem Y and you have to output whatever answer you get back from that subroutine. For example, you can reduce the problem of "is this number even?" to the problem of "is this number odd?" by adding one to the number and outputting whether the resulting number is odd.
As a non-example of a mapping reduction, consider these two problems: first, the problem "is every boolean in this list true?," and second, the problem "is some boolean in this list false?" If you have a solver for the second problem, you can use it to solve the first by running the solver for the second problem and outputting the opposite result: a list of booleans has some element that's false if and only if it's not the case that every element of the list is true. However, this reduction isn't a mapping reduction because we're flipping the result produced by the subroutine.
A different type of reduction that's often used is the Turing reduction. A Turing reduction from problem X to problem Y is the following:
Build an algorithm that solves problem X assuming that there is a magic black box that always solves problem Y.
All mapping reductions are Turing reductions, but not the other way around. The above reduction from "is everything true?" to "is something false" is not a mapping reduction, but it is a Turing reduction because you can use the subroutine for "is something false?" to learn whether or not the list contains any false values, then can output the opposite.
Another major difference between mapping reductions and Turing reductions is that in a Turing reduction, you can make multiple calls to the subroutine that solves problem Y, not just one.
You can think of both quicksort and median-of-medians as algorithms that use partitioning as a subroutine. In quicksort, that subroutine does all the heavy lifting required to sort everything, and in median-of-medians it does one of the essential steps to shrink down the input. Since both algorithms make multiple calls to the subroutine, you can think of them as Turing-style reductions. Quicksort is a reduction from sorting to partitioning, while median-of-medians is a reduction from selection to partitioning.
Hope this helps!

I don't think either can be reduced to the other (in any meaningful way, anyway). You could use the median of medians to choose the pivot for a Quicksort (but nearly nobody actually does). A Quicksort still has to carry out some other steps based on the pivot element though (specifically, partitioning the data based on the pivot).
Likewise, median of medians can't be reduced to Quicksort because a Quicksort does extra work that (among other things) prevents it from meeting the complexity guarantee of the median of medians.

Related

0-1 Knapsack with Large Weights and Values: How much faster is recursive dp than iterative dp?

Suppose I wish to solve 0-1 knapsack but both weights and values of elements can get large. (Weight[i] < 1e5 and Value[i] < 1e9)
Also, number of items is ~2000 and size of knapsack is ~4e6.
Obviously dp[index][weight] exceeds memory and time expectations.
However, somewhat magically memoization dp (values stored in std::map or std::unordered_map) works well. I can see why it may be bit faster: after all, in recursive dp, we compute only the states we need.
But, is it actually significantly faster? In other words, the usual computation requires 8e9 operations, but how much speedup can we expect, on average, by computing dp this way?
Since computing a state takes constant time, the question boils down to - how many (expected / on average) states can be there to compute? (normally it's 8e9)
You can assume:
We use std::unordered_map.
Our hash function works well enough.
Recursive Overheads can be ignored.
Thanks!
In the worst case, the recursive version can compute nearly as many states as the iterative version. Since you've implemented the recursive version without exceeding your available memory, you are pretty far from the worst case.
In special cases, the recursive version can be faster than the naive iterative DP. For example, if all or most of your weights have a large common factor, then that factor will divide the number of states that need to be computed. If you implement the iterative version so that it only considers accessible weights, then it will see this speed-up as well.

Algorithm to solve linear equation system without explicitly spelling out the matrix components

I have a linear function(n inputs -> n outputs), and using special structure of the function(some DP-like algorithm), I can evaluate the output in O(n) time, rather than O(n^2) time. Now, given some output values, I need to find the input that evaluates to the output.
I could spell out the matrix components(by evaluating the linear function with n basis inputs) and use some algorithms like LU decomposition, but that would take O(n^3) time to calculate. Is there a faster algorithm, exploiting the structure of the linear function?
(Since the linear function is not symmetric, Conjugate Gradient method could not be used.)
I need exact solutions, where n is small(n=10~20), but I need to do this kind of calculation hundreds of thousands of times in a second.
From code-design point of view, it would be better if the algorithm did not require transpose of the linear function. (Although at the cost of more code and more debugging, it is possible to provide transpose function with O(n) time complexity.)
Have you considered GMRES? You mentioned that you're looking for exact solutions, however you can get within machine precision error reasonably quickly.
I can evaluate the output in O(n) time, rather than O(n^2) time.
You can use a linear operator to take advantage of this, for example with the GMRES in scipy implementation, A can be a LinearOperator. A linear operator is just a function that evaluates Ax, which is your "evaluate the output" step.
Otherwise, short of an ad hoc solution, I'm not familiar with any exact methods that can be accelerated with linear operators, so I'd need to know more about your problem, eg is your matrix banded?

selection algorithm for median

I was trying to understand the selection algorithm for finding the median. I have pasted the psuedo code below.
SELECT(A[1 .. n], k):
if n<=25
use brute force
else
m = ceiling(n/5)
for i=1 to m
B[i]=SELECT(A[5i-4 .. 5i], 3)
mom=SELECT(B[1 ..m], floor(m/2))
r = PARTITION(A[1 .. n],mom)
if k < r
return SELECT(A[1 .. r-1], k)
else if k > r
return SELECT(A[r +1 .. n], k-r)
else
return mom
i have a very trivial doubt. I was wondering what the author means by brute force written above for i<=25. Is it that he will compare elements one by one with every other element and see if its the kth largest or something else.
The code must come from here.
A brute force algorithm can be any simple and stupid algorithm. In your example, you can sort the 25 elements and find the middle one. This is simple and stupid compared to the selection algorithm since sorting takes O(nlgn) while selection takes only linear time.
A brute force algorithm is often good enough when n is small. Besides, it is easier to implement. Read more about brute force here.
Common wisdom is that Quicksort is slower than insertion sort for small inputs. Therefore many implementations switch to insertion sort at some threshold.
There is a reference to this practice in the Wikipedia page on Quicksort.
Here's an example of commercial mergesort code that switches to insertion sort for small inputs. Here the threshold is 7.
The "brute force" almost certainly refers to the fact that the code here is using the same practice: insertion sort followed by picking the middle element(s) for the median.
However I've found in practice that the common wisdom is not generally true. When I've run benchmarks, the switch has either very little positive effect or negative. That was for Quicksort. In the Parition algorithm, it's more likely ot be negative because one side of the partition is thrown away at each step, so there is less time spent on small inputs. This is verified in #Dennis's response to this SO question.

Is linear-time reduction symmetric?

If a problem X reduces to a problem Y is the opposite reduction also possible? Say
X = Given an array tell if all elements are distinct
Y = Sort an array using comparison sort
Now, X reduces to Y in linear time i.e. if I can solve Y, I can solve X in linear time. Is the reverse always true? Can I solve Y, given I can solve X? If so, how?
By reduction I mean the following:
Problem X linear reduces to problem Y if X can be solved with:
a) Linear number of standard computational steps.
b) Constant calls to subroutine for Y.
Given the example above:
You can determine if all elements are distinct in O(N) if you back them up with a hash table. Which allows you to check existence in O(1) + the overhead of the hash function (which generally doesn't matter). IF you are doing a non-comparison based sort:
sorting algorithm list
Specialized sort that is linear:
For simplicity, assume you're sorting a list of natural numbers. The sorting method is illustrated using uncooked rods of spaghetti:
For each number x in the list, obtain a rod of length x. (One practical way of choosing the unit is to let the largest number m in your list correspond to one full rod of spaghetti. In this case, the full rod equals m spaghetti units. To get a rod of length x, simply break a rod in two so that one piece is of length x units; discard the other piece.)
Once you have all your spaghetti rods, take them loosely in your fist and lower them to the table, so that they all stand upright, resting on the table surface. Now, for each rod, lower your other hand from above until it meets with a rod--this one is clearly the longest! Remove this rod and insert it into the front of the (initially empty) output list (or equivalently, place it in the last unused slot of the output array). Repeat until all rods have been removed.
So given a very specialized case of your problem, your statement would hold. This will not hold in the general case though, which seems to be more what you are after. It is very similar to when people think they have solved TSP, but have instead created a constrained version of the general problem that is solvable using a special algorithm.
Suppose I can solve a problem A in constant time O(1) but problem B has a best case exponential time solution O(2^n). It is likely that I can come up with an insanely complex way of solving problem A in O(2^n) ("reducing" problem A to B) as well but if the answer to your question was "YES", I should then be able to make all exceedingly difficult problems solvable in O(1). Surely, that cannot be the case!
Assuming I understand what you mean by reduction, let's say that I have a problem that I can solve in O(N) using an array of key/value pairs, that being the problem of looking something up from a list. I can solve the same problem in O(1) by using a Dictionary.
Does that mean I can go back to my first technique, and use it to solve the same problem in O(1)?
I don't think so.

Why is Binary Search a divide and conquer algorithm?

I was asked if a Binary Search is a divide and conquer algorithm at an exam. My answer was yes, because you divided the problem into smaller subproblems, until you reached your result.
But the examinators asked where the conquer part in it was, which I was unable to answer. They also disapproved that it actually was a divide and conquer algorithm.
But everywhere I go on the web, it says that it is, so I would like to know why, and where the conquer part of it is?
The book:
Data Structures and Algorithm Analysis in Java (2nd Edition), by Mark Allen Weiss
Says that a D&C algorithm should have two disjoint recursive calls, just like QuickSort does.
Binary Search does not have this, even though it can be implemented recursively.
I think it is not divide and conquer, see first paragraph in http://en.wikipedia.org/wiki/Divide_and_conquer_algorithm
recursively breaking down a problem into two or more sub-problems
which are then combined to give a solution
In binary search there is still only one problem which does just reducing data by half every step, so no conquer (merging) phase of the results is needed.
It isn't.
To complement #Kenci's post, DnC algorithms have a few general/common properties; they:
divide the original problem instance into a set of smaller sub-instances of itself;
independently solve each sub-instance;
combine smaller/independent sub-instance solutions to build a single solution for the larger/original instance
The problem with Binary Search is that it does not really even generate a set of independent sub-instances to be solved, as per step 1; it only simplifies the original problem by permanently discarding sections it's not interested in. In other words, it only reduces the problem's size and that's as far as it ever goes.
A DnC algorithm is supposed to not only identify/solve the smaller sub-instances of the original problem independently of each other, but also use that set of partial independent solutions to "build up" a single solution for the larger problem instance as a whole.
The book Fundamentals of Algorithmics, G. Brassard, P. Bratley says the following (bold my emphasis, italics in original):
It is probably the simplest application of divide-and-conquer, so simple in fact that strictly speaking this is an application of simplification rather than divide-and-conquer: the solution to any sufficiently large instance is reduced to that of a single smaller one, in this case of half size.
Section 7.3 Binary Search on p.226.
In a divide and conquer strategy :
1.Problem is divided into parts;
2.Each of these parts is attacked/solved independently, by applying the algorithm at hand (mostly recursion is used for this purpose) ;
3.And then the solutions of each partition/division and combined/merged together to arrive at the final solution to the problem as a whole (this comes under conquer)
Example, Quick sort, merge sort.
Basically, the binary search algorithm just divides its work space(input (ordered) array of size n) into half in each iteration. Therefore it is definitely deploying the divide strategy and as a result, the time complexity reduces down to O(lg n).So,this covers up the "divide" part of it.
As can be noticed, the final solution is obtained from the last comparison made, that is, when we are left with only one element for comparison.
Binary search does not merge or combine solution.
In short, binary search divides the size of the problem (on which it has to work) into halves but doesn't find the solution in bits and pieces and hence no need of merging the solution occurs!
I know it's a bit too lengthy but i hope it helps :)
Also you can get some idea from : https://www.khanacademy.org/computing/computer-science/algorithms/binary-search/a/running-time-of-binary-search
Also i realised just now that this question was posted long back!
My bad!
Apparently some people consider binary search a divide-and-conquer algorithm, and some are not. I quickly googled three references (all seem related to academia) that call it a D&C algorithm:
http://www.cs.berkeley.edu/~vazirani/algorithms/chap2.pdf
http://homepages.ius.edu/rwisman/C455/html/notes/Chapter2/DivConq.htm
http://www.csc.liv.ac.uk/~ped/teachadmin/algor/d_and_c.html
I think it's common agreement that a D&C algorithm should have at least the first two phases of these three:
divide, i.e. decide how the whole problem is separated into sub-problems;
conquer, i.e. solve each of the sub-problems independently;
[optionally] combine, i.e. merge the results of independent computations together.
The second phase - conquer - should recursively apply the same technique to solve the subproblem by dividing into even smaller sub-sub-problems, and etc. In practice, however, often some threshold is used to limit the recursive approach, as for small size problems a different approach might be faster. For example, quick sort implementations often use e.g. bubble sort when the size of an array portion to sort becomes small.
The third phase might be a no-op, and in my opinion it does not disqualify an algorithm as D&C. A common example is recursive decomposition of a for-loop with all iterations working purely with independent data items (i.e. no reduction of any form). It might look useless at glance, but in fact it's very powerful way to e.g. execute the loop in parallel, and utilized by such frameworks as Cilk and Intel's TBB.
Returning to the original question: let's consider some code that implements the algorithm (I use C++; sorry if this is not the language you are comfortable with):
int search( int value, int* a, int begin, int end ) {
// end is one past the last element, i.e. [begin, end) is a half-open interval.
if (begin < end)
{
int m = (begin+end)/2;
if (value==a[m])
return m;
else if (value<a[m])
return search(value, a, begin, m);
else
return search(value, a, m+1, end);
}
else // begin>=end, i.e. no valid array to search
return -1;
}
Here the divide part is int m = (begin+end)/2; and all the rest is the conquer part. The algorithm is explicitly written in a recursive D&C form, even though only one of the branches is taken. However, it can also be written in a loop form:
int search( int value, int* a, int size ) {
int begin=0, end=size;
while( begin<end ) {
int m = (begin+end)/2;
if (value==a[m])
return m;
else if (value<a[m])
end = m;
else
begin = m+1;
}
return -1;
}
I think it's quite a common way to implement binary search with a loop; I deliberately used the same variable names as in the recursive example, so that commonality is easier to see. Therefore we might say that, again, calculating the midpoint is the divide part, and the rest of the loop body is the conquer part.
But of course if your examiners think differently, it might be hard to convince them it's D&C.
Update: just had a thought that if I were to develop a generic skeleton implementation of a D&C algorithm, I would certainly use binary search as one of API suitability tests to check whether the API is sufficiently powerful while also concise. Of course it does not prove anything :)
The Merge Sort and Quick Sort algorithms use the divide and conquer technique (because there are 2 sub-problems) and Binary Search comes under decrease and conquer (because there is 1 sub-problem).
Therefore, Binary Search actually uses the decrease and conquer technique and not the divide and conquer technique.
Source: https://www.geeksforgeeks.org/decrease-and-conquer/
Binary search is tricky to describe with divide-and-conquer because the conquering step is not explicit. The result of the algorithm is the index of the needle in the haystack, and a pure D&C implementation would return the index of the needle in the smallest haystack (0 in the one-element list) and then recursively add the offsets in the larger haystacks that were divided in the divison step.
Pseudocode to explain:
function binary_search has arguments needle and haystack and returns index
if haystack has size 1
return 0
else
divide haystack into upper and lower half
if needle is smaller than smallest element of upper half
return 0 + binary_search needle, lower half
else
return size of lower half + binary_search needle, upper half
The addition (0 + or size of lower half) is the conquer part. Most people skip it by providing indices into a larger list as arguments, and thus it is often not readily available.
The divide part is of course dividing the set into halves.
The conquer part is determining whether and on what position in the processed part there is a searched element.
Dichotomic in computer science refers to choosing between two antithetical choices, between two distinct alternatives. A dichotomy is any splitting of a whole into exactly two non-overlapping parts, meaning it is a procedure in which a whole is divided into two parts. It is a partition of a whole (or a set) into two parts (subsets) that are:
1. Jointly Exhaustive: everything must belong to one part or the other, and
2. Mutually Exclusive: nothing can belong simultaneously to both parts.
Divide and conquer works by recursively breaking down a problem into two or more sub-problems of the same type, until these become simple enough to be solved directly.
So the binary search halves the number of items to check with each iteration and determines if it has a chance of locating the "key" item in that half or moving on to the other half if it is able to determine keys absence. As the algorithm is dichotomic in nature so the binary search will believe that the "key" has to be in one part until it reaches the exit condition where it returns that the key is missing.
Divide and Conquer algorithm is based on 3 step as follows:
Divide
Conquer
Combine
Binary Search problem can be defined as finding x in the sorted array A[n].
According to this information:
Divide: compare x with middle
Conquer: Recurse in one sub array. (Finding x in this array)
Combine: it is not necessary.
A proper divide and conquer algorithm will require both parts to be processed.
Therefore, many people will not call binary-search a divide and conquer algorithm, it does divide the problem, but discards the other half.
But most likely, your examiners just wanted to see how you argue. (Good) exams aren't about the facts, but about how you react when the challenge goes beyond the original material.
So IMHO the proper answer would have been:
Well, technically, it consists only of a divide step, but needs to conquer only half of the original task then, the other half is trivially done already.
BTW: there is a nice variation of QuickSort, called QuickSelect, which actually exploits this difference to obtain an on average O(n) median search algorithm. It's like QuickSort - but descends only into the half it is interested in.
Binary Search is not a divide and conquer approach. It is a decrease and conquer approach.
In divide and conquer approach, each subproblem must contribute to the solution but in binary search, all subdivision does not contribute to the solution. we divide into two parts and discard one part because we know that the solution does not exist in this part and look for the solution only in one part.
The informal definition is more or less: Divide the problem into small problems. Then solve them and put them together (conquer). Solving is in fact deciding where to go next (left, right, element found).
Here a quote from wikipedia:
The name "divide and conquer" is sometimes applied also to algorithms that reduce each problem to only one subproblem, such as the binary search algorithm for finding a record in a sorted list.
This states, it's NOT [update: misread this phrase:)] only one part of divide and conquer.
Update:
This article made it clear for me. I was confused since the definition says you have to solve every sub problem. But you solved the sub problem if you know you don't have to keep on searching..
The Binary Search is a divide and conquer algorithm:
1) In Divide and Conquer algorithms, we try to solve a problem by solving a smaller sub problem (Divide part) and use the solution to build the solution for our bigger problem(Conquer).
2) Here our problem is to find an element in the sorted array. We can solve this by solving a similar sub problem. (We are creating sub problems here based on a decision that the element being searched is smaller or bigger than the middle element). Thus once we know that the element can not exist surely in one half, we solve a similar sub-problem in the the other half.
3) This way we recurse.
4) The conquer part here is just returning the value returned by the sub problem to the top the recursive tree
I think it is Decrease and Conquer.
Here is a quote from wikipedia.
"The name decrease and conquer has been proposed instead for the
single-subproblem class"
http://en.wikipedia.org/wiki/Divide_and_conquer_algorithms#Decrease_and_conquer
According to my understanding, "Conquer" part is at the end when you find the target element of the Binary search. The "Decrease" part is reducing the search space.
Binary Search and Ternary Search Algorithms are based on Decrease and Conquer technique. Because, you do not divide the problem, you actually decrease the problem by dividing by 2(3 in ternary search).
Merge Sort and Quick Sort Algorithms can be given as examples of Divide and Conquer technique. You divide the problem into two subproblems and use the algorithm for these subproblems again to sort an array. But, you discard the half of array in binary search. It means you DECREASE the size of array, not divide.
No, binary search is not divide and conquer. Yes, binary search is decrease and conquer. I believe divide and conquer algorithms have an efficiency of O(n log(n)) while decrease and conquer algorithms have an efficiency of O(log(n)). The difference being whether or not you need to evaluate both parts of the split in data or not.

Resources