Initialize:
max_so_far = 0
max_ending_here = 0
Loop for each element of the array
(a) max_ending_here = max_ending_here + a[i]
(b) if(max_ending_here < 0)
max_ending_here = 0
(c) if(max_so_far < max_ending_here)
max_so_far = max_ending_here
return max_so_far
Can anyone help me in understanding the optimal substructure and overlapping problem(bread and butter of DP) i the above algo?
According to this definition of overlapping subproblems, the recursive formulation of Kadane's algorithm (f[i] = max(f[i - 1] + a[i], a[i])) does not exhibit this property. Each subproblem would only be computed once in a naive recursive implementation.
It does however exhibit optimal substructure according to its definition here: we use the solution to smaller subproblems in order to find the solution to our given problem (f[i] uses f[i - 1]).
Consider the dynamic programming definition here:
In mathematics, computer science, and economics, dynamic programming is a method for solving complex problems by breaking them down into simpler subproblems. It is applicable to problems exhibiting the properties of overlapping subproblems1 and optimal substructure (described below). When applicable, the method takes far less time than naive methods that don't take advantage of the subproblem overlap (like depth-first search).
The idea behind dynamic programming is quite simple. In general, to solve a given problem, we need to solve different parts of the problem (subproblems), then combine the solutions of the subproblems to reach an overall solution. Often when using a more naive method, many of the subproblems are generated and solved many times. The dynamic programming approach seeks to solve each subproblem only once, thus reducing the number of computations
This leaves room for interpretation as to whether or not Kadane's algorithm can be considered a DP algorithm: it does solve the problem by breaking it down into easier subproblems, but its core recursive approach does not generate overlapping subproblems, which is what DP is meant to handle efficiently - so this would put it outside DP's specialty.
On the other hand, you could say that it is not necessary for the basic recursive approach to lead to overlapping subproblems, but this would make any recursive algorithm a DP algorithm, which would give DP a much too broad scope in my opinion. I am not aware of anything in the literature that definitely settles this however, so I wouldn't mark down a student or disconsider a book or article either way they labeled it.
So I would say that it is not a DP algorithm, just a greedy and / or recursive one, depending on the implementation. I would label it as greedy from an algorithmic point of view for the reasons listed above, but objectively I would consider other interpretations just as valid.
Note that I derived my explanation from this answer. It demonstrates how Kadane’s algorithm can be seen as a DP algorithm which has overlapping subproblems.
Identifying subproblems and recurrence relations
Imagine we have an array a from which we want to get the maximum subarray. To determine the max subarray that ends at index i the following recursive relation holds:
max_subarray_to(i) = max(max_subarray_to(i - 1) + a[i], a[i])
In order to get the maximum subarray of a we need to compute max_subarray_to() for each index i in a and then take the max() from it:
max_subarray = max( for i=1 to n max_subarray_to(i) )
Example
Now, let's assume we have an array [10, -12, 11, 9] from which we want to get the maximum subarray. This would be the work required running Kadane's algorithm:
result = max(max_subarray_to(0), max_subarray_to(1), max_subarray_to(2), max_subarray_to(3))
max_subarray_to(0) = 10 # base case
max_subarray_to(1) = max(max_subarray_to(0) + (-12), -12)
max_subarray_to(2) = max(max_subarray_to(1) + 11, 11)
max_subarray_to(3) = max(max_subarray_to(2) + 9, 49)
As you can see, max_subarray_to() is evaluated twice for each i apart from the last index 3, thus showing that Kadane's algorithm does have overlapping subproblems.
Kadane's algorithm is usually implemented using a bottom up DP approach to take advantage of the overlapping subproblems and to only compute each subproblem once, hence turning it to O(n).
Related
I have obtained a proof that would discredit a generally held idea regarding the 0/1 knapsack problem and I'm really having a hard time convincing my self I am right because I couldn't find any thing any where to support my claims, so I am going to first state my claims and then prove them and I would appreciate anyone to try to substantiate my claims further or disproof them. Any collaboration is appreciated.
Assertions:
The size of the bnb (branch and bound) algorithm for solving the knapsack problem is not independent of the K (capacity of the knapsack).
The size of bnb tree complete space is always of O(NK) with N being the number of items and not O(2^N)
The bnb algorithm is always better than the standard dynamic programming approach both in time and space.
Pre-assumptions: The bnb algorithm prone the invalid nodes (if the remaining capacity is less than the weight of current item, we are not going to extend it. Also, the bnb algorithm is done in a depth-first manner.
Sloppy Proof:
Here is the recursive formula for solving the knapsack problem:
Value(i,k) = max (Value(i-1,k) , Value(n-1 , k-weight(i)) + value(i)
however if k < weight(i): Value(i,k) = Value(i-1,k)
Now imagine this example:
K = 9
N = 3
V W:
5 4
6 5
3 2
Now here is the Dynamic solution and table for this problem:
Now imagine regardless of whether it is a good idea or not we want to do this using only the recursive formula through memoization and not with the table, with something like a map/dictionary or a simple array to store the visited cells. For solving this problem using memoization we should solve the denoted cells:
Now this is exactly like the tree we would obtain using the bnb approach:
and now for the sloppy proofs:
Memoization and bnb tree have the same amount of nodes
Memoization nodes is dependent of the table size
Table size is dependent of N and K
Therefore bnb is not independent of K
Memoization space is bounded by NK i.e. O(NK)
Therefore bnb tree complete space (or the space if we do the bnb in a breadth first manner) is always of O(NK) and not O(N^2) because the whole tree is not going to be constructed and it would be exactly like the momization.
Memoization has better space than the standard dynamic programming.
bnb has better space than the dynamic programming (even if done in breadth first)
The simple bnb without relaxation (and just eliminating the infeasible nodes) would have better time than memoization (memoization has to search in the look up table and even if the look up was negligible they would still be the same.)
If we disregard the look up search of memoization, it is better than dynamic.
Therefore bnb algorithm is always better than dynamic both in time and space.
Questions:
If by any mean my proofs are correct some questions would arise that are interesting:
Why bother with dynamic programming? In my experience the best thing you could do in dp knapsack is to have the last two columns and you can improve it further to one column if you fill it bottom to top, and it would have O(K) space but still can't (if the above assertions are correct) beat the bnb approach.
Can we still say bnb is better if we integrate it with relaxation pruning (with regard to time)?
ps: Sorry for the long long post!
Edit:
Since two of the answers are focused on memoization, I just want to clarify that I'm not focused on this at all! I just used memoization as a technique to prove my assertions. My main focus is Branch and Bound technique vs dynamic programming, here is a complete example of another problem, solved by bnb + relaxation (source: Coursera - Discrete Optimization) :
I think there is a misunderstanding from your side, that the dynamic programming is the state-of-the art solution for the knapsack problem. This algorithm is taught at universities because it is an easy and nice example for dynamic programming and pseudo-polynomial time algorithms.
I have no expertise in the field and don't know what is the state-of-the art now, but branch-and-bound approaches have been used for quite some time to solve the knapsack-problem: The book Knapsak-Problems by Martello and Toth is already pretty old but treats the branch-and-bound pretty extensively.
Still, this is a great observation from your side, that the branch and bound approach can be used for knapsack - alas, you were born too late to be the first to have this idea:)
There are some points in your proof which I don't understand and which need more explanation in my opinion:
You need memoization, otherwise your tree would have O(2^N) nodes (there will be obviously such a case otherwise knapsack would not be NP-hard). I don't see anything in your proof, that assures that the memoization memory/computation steps are less than O(NK).
Dynamical programming needs only O(K) memory-space, so I don't see why you could claim "bnb algorithm is always better than dynamic both in time and space".
Maybe your claims are true, but I'm not able to see it the way the proof goes now.
The other problem is the definition of "better". Is branch-and-bound approach better if it is better for most of the problems or the common problems or does it has to be better for the wost-case (which would not play any role in the real life)?
The book I have linked to has also some comparisons for the running times of the algorithms. The dynamic programming based algorithms (clearly more complex as the one taught at school) are even better for some kind of problems - see section 2.10.1. Not bad for a total joke!
First of all, since you are applying memorization, you are still doing DP. That's basically the definition of DP: recursion + memorization. And that is also good. Without memorization your computation costs would explode. Just imagine if two items both have weight 2 and a third and a fourth have weight 1. They all end up at the same node in the tree, you would have to do the computation multiple times and you'll end up with exponential running time.
The main difference is the order of computation. The way of computing the entire matrix is called "bottom-up DP", since you start with (0,0) and work yourself upwards. Your way (the tree approach) is called "top-down DP", since you start with the goal and work yourself down the tree. But they are both using dynamic programming.
Now to your questions:
You are overestimating how much you really save. N = 3 is a pretty small example. I quickly tried a bigger example, with N = 20, K=63 (which is still pretty small) and random values and random weights. This is the first picture that I've generated:
values: [4, 10, 9, 1, 1, 2, 1, 2, 6, 4, 8, 9, 8, 2, 8, 8, 4, 10, 2, 6]
weights: [6, 4, 1, 10, 1, 2, 9, 9, 1, 6, 2, 3, 10, 7, 2, 4, 10, 9, 8, 2]
111111111111111111111111111111111111111111111111111111111111111
111111111111111111111111111111111111111111111111111111111111111
111111111111111111111111111111111111111111111111111111111111111
111111111111111111111111111111111111111111111111111111111111111
111111111111111111111111111111111111111111111111111111111111111
111111111111111111111111111111111111111111111111111111111111111
111111111111111111111111111111111111111111111111111111111111111
111111111111111111111111111111111111111111111111111111111111111
111111111111111111111111111111111111111111111111111111111111111
011111111111111111111111111111111111111111111111111111111111101
000001011111111111111111111111111111111111111111111111111111101
000000010111111111111111111111111111111111111111111111111111101
000000000010101011111111111111111111111111111111111111111010101
000000000000000000001010101111111111111111111111111111111010101
000000000000000000000000000101010101111111111111111111101010101
000000000000000000000000000001010101011111111111111111101010101
000000000000000000000000000000000101000001111100001111100000101
000000000000000000000000000000000000000000010100000111100000101
000000000000000000000000000000000000000000000000000010100000101
000000000000000000000000000000000000000000000000000000000000101
000000000000000000000000000000000000000000000000000000000000001
This picture is a transposed version of your displayed matrix. Rows represent the i values (first i elements in the array), and the cols represent the k values (allowed weights). The 1s are the positions in the DP matrix that you will visit during your tree-approach. Of course you'll see a lot of 0s at the bottom of the matrix, but you will visit every position the the upper half. About 68% of the positions in the matrix are visited. A bottom-up DP solution will be faster in such a situation. Recursion calls are slower, since you have to allocate a new stack frame for each recursive call. A speedup of 2x with loops instead of recursive calls is not untypical, and this would already be enough to make the bottom up approach faster. And we haven't even talked about the memorization costs of the tree approach yet.
Notice, that I haven't used actual bnb here. I'm not quite sure how you would do the bound-part, since you actually only know the value of a node once you compute it by visiting its children.
With my input data, the bottom-up approach is clearly a winner. But that doesn't mean that your approach is bad. Quite the opposite. It can actually be quite good. It all depends on the input data. Let's just imagine that K = 10^18 and all your weights are about 10^16. The bottom-up approach would not even find enough memory to allocate the matrix, while your approach will succeed in no time.
However, you probably could improve your version by performing A* instead of bnb. You can estimate the best value for each node (i, k) with int(k / max(weight[1..i]) * min(values[1..i]) and prune a lot of nodes using this heuristic.
In practice dynamic programming can be better for integer 0/1 knapsack because:
No recursion means you can never run into a stack overflow
No need to do a lookup search for each node, so often faster
As you note, storing the last two columns means that the memory requirement is lower
The code is simpler (no need for a memoization table)
I'm having trouble understanding the dynamic programming solution to the text justification problem as specified in the MIT open courseware lecture here. Some notes from that lecture are here, and page 3 of the notes is what I am referring to.
I thought that Dynamic Programming meant you memoize some of the computations so that you don't need to recompute, thus saving you time, but in the algorithm given in the lecture, I don't see any use of memoization, just a whole bunch of deep recursive calls, i.e. the main function is this:
DP[i] = min(badness (i, j) + DP[j] for j in range (i + 1, n + 1))
DP[n] = 0
where badness is a function that determines the the amount of unused space after subtracting the length of the words from the line length. To me it looks like this algorithm calculates all possible "badness" calculations and chooses the smallest one, which seems like brute force to me. Where is the advantage Dynamic Programming usually gives us by memoizing past calculations so we don't have to recompute?
If you memoize the results, you don't have to compute each DP[i] several times.
That is, DP[0] "calls" DP[2] for example, but so does DP[1]. In the second time DP[2] is called, it won't be necessary to compute it again, you can just return the memoized value.
This also makes it easy to verify a polynomial upper bound for this algorithm. Since each DP[i] will perform O(n) operations, and there are n of them, the overall algorithm is O(n^2), assuming, of course, that badness(i, j) is O(1).
Is there any way of finding the longest common subsequence of two sequences in O(NlogN) time?
I read somewhere that there is a way to achieve this using binary search.
I know the dp approach that takes O(N2) time.
For the general case, the O(N^2) dynamic programming algorithm is the best you can do. However, there exist better algorithms in some special cases.
Alphabet size is bounded
This is a very common situation. Sequences consisting of letters from some alphabet (e.g. English) lie in this category. For this case, the O(N*M) algorithm can be optimised to get an O(N^2/logN) with method of four Russians. I don't know how exactly, you can search for it.
Both sequences consist of distinct elements
An example problem is "Given two permutations of numbers from 1 to N, find their LCS". This one can be solved in O(N*logN). Let the sequences be A and B.
Define a sequence C. C[i] is the index of B[i] in A. (A[C[i]] = B[i])
LCS of A and B is the longest increasing subsequence of C.
The dynamic programming approach, which is O(n2) for general case. For certain other cases, there are lower-complexity algorithms:
For a fixed alphabet size (which doesn't grow with n), there's the Method of Four Russians which brings the time down to O(n2/log n) (see here).
See here another further optimized case.
Assuming Exponential Time Hypothesis (which is stricter than P is not equal to NP, but is still widely believed to be true), it is not possible to do it in time O(N^{2 - eps}) for any positive constant eps, see "Quadratic Conditional Lower Bounds for String Problems and Dynamic Time Warping" by Karl Bringmann and Marvin Kunnemann (pre-print on arXiv is available).
Roughly speaking, it means that the general case of this problem cannot be solved in time better than something like O(N^2/log N), so if you want faster algorithms you have to consider additional constraints (some properties of the strings) or look for approximate solution.
The longest common subsequence between two sequences is essentially n-squared.
Masek and Patterson (1980) made a minor improvement to n-squared / log n using the so-called "Four Russians" technique.
In most cases the additional complexity introduced by such convoluted approaches is not justified by the small gains. For practical purposes you can consider the n-squared approach as the reasonable optimum in typical applications.
vector <int> LIS;
int LongestIncreasingSubsequence(int n){
if(!n) return 0;
LIS.emplace_back(arr[0]);
for(int i = 1; i < n; i++){
if(arr[i] > LIS.back()) LIS.emplace_back(arr[i]);
else *lower_bound(LIS.begin(), LIS.end(), arr[i]) = arr[i];
}
return LIS.size();
}
I'm learning dynamic programming and I've been having a great deal of trouble understanding more complex problems. When given a problem, I've been taught to find a recursive algorithm, memoize the recursive algorithm and then create an iterative, bottom-up version. At almost every step I have an issue. In terms of the recursive algorithm, I write different different ways to do recursive algorithms, but only one is often optimal for use in dynamic programming and I can't distinguish what aspects of a recursive algorithm make memoization easier. In terms of memoization, I don't understand which values to use for indices. For conversion to a bottom-up version, I can't figure out which order to fill the array/double array.
This is what I understand:
- it should be possible to split the main problem to subproblems
In terms of the problem mentioned, I've come up with a recursive algorithm that has these important lines of code:
int optionOne = values[i] + find(values, i+1, limit - values[i]);
int optionTwo = find(values, i+1, limit);
If I'm unclear or this is not the correct qa site, let me know.
Edit:
Example: Given array x: [4,5,6,9,11] and max value m: 20
Maximum subsequence in x under or equal to m would be [4,5,11] as 4+5+11 = 20
I think this problem is NP-hard, meaning that unless P = NP there isn't a polynomial-time algorithm for solving the problem.
There's a simple reduction from the subset-sum problem to this problem. In subset-sum, you're given a set of n numbers and a target number k and want to determine whether there's a subset of those numbers that adds up to exactly k. You can solve subset-sum with a solver for your problem as follows: create an array of the numbers in the set and find the largest subsequence whose sum is less than or equal to k. If that adds up to exactly k, the set has a subset that adds up to k. Otherwise, it does not.
This reduction takes polynomial time, so because subset-sum is NP-hard, your problem is NP-hard as well. Therefore, I doubt there's a polynomial-time algorithm.
That said - there is a pseudopolynomial-time algorithm for subset-sum, which is described on Wikipedia. This algorithm uses DP in two variables and isn't strictly polynomial time, but it will probably work in your case.
Hope this helps!
What is the difference between Divide and Conquer Algorithms and Dynamic Programming Algorithms? How are the two terms different? I do not understand the difference between them.
Please take a simple example to explain any difference between the two and on what ground they seem to be similar.
Divide and Conquer
Divide and Conquer works by dividing the problem into sub-problems, conquer each sub-problem recursively and combine these solutions.
Dynamic Programming
Dynamic Programming is a technique for solving problems with overlapping subproblems. Each sub-problem is solved only once and the result of each sub-problem is stored in a table ( generally implemented as an array or a hash table) for future references. These sub-solutions may be used to obtain the original solution and the technique of storing the sub-problem solutions is known as memoization.
You may think of DP = recursion + re-use
A classic example to understand the difference would be to see both these approaches towards obtaining the nth fibonacci number. Check this material from MIT.
Divide and Conquer approach
Dynamic Programming Approach
Dynamic Programming and Divide-and-Conquer Similarities
As I see it for now I can say that dynamic programming is an extension of divide and conquer paradigm.
I would not treat them as something completely different. Because they both work by recursively breaking down a problem into two or more sub-problems of the same or related type, until these become simple enough to be solved directly. The solutions to the sub-problems are then combined to give a solution to the original problem.
So why do we still have different paradigm names then and why I called dynamic programming an extension. It is because dynamic programming approach may be applied to the problem only if the problem has certain restrictions or prerequisites. And after that dynamic programming extends divide and conquer approach with memoization or tabulation technique.
Let’s go step by step…
Dynamic Programming Prerequisites/Restrictions
As we’ve just discovered there are two key attributes that divide and conquer problem must have in order for dynamic programming to be applicable:
Optimal substructure — optimal solution can be constructed from optimal solutions of its subproblems
Overlapping sub-problems — problem can be broken down into subproblems which are reused several times or a recursive algorithm for the problem solves the same subproblem over and over rather than always generating new subproblems
Once these two conditions are met we can say that this divide and conquer problem may be solved using dynamic programming approach.
Dynamic Programming Extension for Divide and Conquer
Dynamic programming approach extends divide and conquer approach with two techniques (memoization and tabulation) that both have a purpose of storing and re-using sub-problems solutions that may drastically improve performance. For example naive recursive implementation of Fibonacci function has time complexity of O(2^n) where DP solution doing the same with only O(n) time.
Memoization (top-down cache filling) refers to the technique of caching and reusing previously computed results. The memoized fib function would thus look like this:
memFib(n) {
if (mem[n] is undefined)
if (n < 2) result = n
else result = memFib(n-2) + memFib(n-1)
mem[n] = result
return mem[n]
}
Tabulation (bottom-up cache filling) is similar but focuses on filling the entries of the cache. Computing the values in the cache is easiest done iteratively. The tabulation version of fib would look like this:
tabFib(n) {
mem[0] = 0
mem[1] = 1
for i = 2...n
mem[i] = mem[i-2] + mem[i-1]
return mem[n]
}
You may read more about memoization and tabulation comparison here.
The main idea you should grasp here is that because our divide and conquer problem has overlapping sub-problems the caching of sub-problem solutions becomes possible and thus memoization/tabulation step up onto the scene.
So What the Difference Between DP and DC After All
Since we’re now familiar with DP prerequisites and its methodologies we’re ready to put all that was mentioned above into one picture.
If you want to see code examples you may take a look at more detailed explanation here where you'll find two algorithm examples: Binary Search and Minimum Edit Distance (Levenshtein Distance) that are illustrating the difference between DP and DC.
The other difference between divide and conquer and dynamic programming could be:
Divide and conquer:
Does more work on the sub-problems and hence has more time consumption.
In divide and conquer the sub-problems are independent of each other.
Dynamic programming:
Solves the sub-problems only once and then stores it in the table.
In dynamic programming the sub-problem are not independent.
sometimes when programming recursivly, you call the function with the same parameters multiple times which is unnecassary.
The famous example Fibonacci numbers:
index: 1,2,3,4,5,6...
Fibonacci number: 1,1,2,3,5,8...
function F(n) {
if (n < 3)
return 1
else
return F(n-1) + F(n-2)
}
Let's run F(5):
F(5) = F(4) + F(3)
= {F(3)+F(2)} + {F(2)+F(1)}
= {[F(2)+F(1)]+1} + {1+1}
= 1+1+1+1+1
So we have called :
1 times F(4)
2 times F(3)
3 times F(2)
2 times F(1)
Dynamic Programming approach: if you call a function with the same parameter more than once, save the result into a variable to directly access it on next time. The iterative way:
if (n==1 || n==2)
return 1
else
f1=1, f2=1
for i=3 to n
f = f1 + f2
f1 = f2
f2 = f
Let's call F(5) again:
fibo1 = 1
fibo2 = 1
fibo3 = (fibo1 + fibo2) = 1 + 1 = 2
fibo4 = (fibo2 + fibo3) = 1 + 2 = 3
fibo5 = (fibo3 + fibo4) = 2 + 3 = 5
As you can see, whenever you need the multiple call you just access the corresponding variable to get the value instead of recalculating it.
By the way, dynamic programming doesn't mean to convert a recursive code into an iterative code. You can also save the subresults into a variable if you want a recursive code. In this case the technique is called memoization. For our example it looks like this:
// declare and initialize a dictionary
var dict = new Dictionary<int,int>();
for i=1 to n
dict[i] = -1
function F(n) {
if (n < 3)
return 1
else
{
if (dict[n] == -1)
dict[n] = F(n-1) + F(n-2)
return dict[n]
}
}
So the relationship to the Divide and Conquer is that D&D algorithms rely on recursion. And some versions of them has this "multiple function call with the same parameter issue." Search for "matrix chain multiplication" and "longest common subsequence" for such examples where DP is needed to improve the T(n) of D&D algo.
I assume you have already read Wikipedia and other academic resources on this, so I won't recycle any of that information. I must also caveat that I am not a computer science expert by any means, but I'll share my two cents on my understanding of these topics...
Dynamic Programming
Breaks the problem down into discrete subproblems. The recursive algorithm for the Fibonacci sequence is an example of Dynamic Programming, because it solves for fib(n) by first solving for fib(n-1). In order to solve the original problem, it solves a different problem.
Divide and Conquer
These algorithms typically solve similar pieces of the problem, and then put them together at the end. Mergesort is a classic example of divide and conquer. The main difference between this example and the Fibonacci example is that in a mergesort, the division can (theoretically) be arbitrary, and no matter how you slice it up, you are still merging and sorting. The same amount of work has to be done to mergesort the array, no matter how you divide it up. Solving for fib(52) requires more steps than solving for fib(2).
I think of Divide & Conquer as an recursive approach and Dynamic Programming as table filling.
For example, Merge Sort is a Divide & Conquer algorithm, as in each step, you split the array into two halves, recursively call Merge Sort upon the two halves and then merge them.
Knapsack is a Dynamic Programming algorithm as you are filling a table representing optimal solutions to subproblems of the overall knapsack. Each entry in the table corresponds to the maximum value you can carry in a bag of weight w given items 1-j.
Divide and Conquer involves three steps at each level of recursion:
Divide the problem into subproblems.
Conquer the subproblems by solving them recursively.
Combine the solution for subproblems into the solution for original problem.
It is a top-down approach.
It does more work on subproblems and hence has more time
consumption.
eg. n-th term of Fibonacci series can be computed in O(2^n) time complexity.
Dynamic Programming involves the following four steps:
1. Characterise the structure of optimal solutions.
2. Recursively define the values of optimal solutions.
3. Compute the value of optimal solutions.
4. Construct an Optimal Solution from computed information.
It is a Bottom-up approach.
Less time consumption than divide and conquer since we make use of the values computed earlier, rather than computing again.
eg. n-th term of Fibonacci series can be computed in O(n) time complexity.
For easier understanding, lets see divide and conquer as a brute force solution and its optimisation as dynamic programming.
N.B. divide and conquer algorithms with overlapping subproblems can only be optimised with dp.
Divide and Conquer
They broke into non-overlapping sub-problems
Example: factorial numbers i.e. fact(n) = n*fact(n-1)
fact(5) = 5* fact(4) = 5 * (4 * fact(3))= 5 * 4 * (3 *fact(2))= 5 * 4 * 3 * 2 * (fact(1))
As we can see above, no fact(x) is repeated so factorial has non overlapping problems.
Dynamic Programming
They Broke into overlapping sub-problems
Example: Fibonacci numbers i.e. fib(n) = fib(n-1) + fib(n-2)
fib(5) = fib(4) + fib(3) = (fib(3)+fib(2)) + (fib(2)+fib(1))
As we can see above, fib(4) and fib(3) both use fib(2). similarly so many fib(x) gets repeated. that's why Fibonacci has overlapping sub-problems.
As a result of the repetition of sub-problem in DP, we can keep such results in a table and save computation effort. this is called as memoization
Divide and Conquer
In this problem is solved in following three steps:
1. Divide - Dividing into number of sub-problems
2. Conquer - Conquering by solving sub-problems recursively
3. Combine - Combining sub-problem solutions to get original problem's solution
Recursive approach
Top Down technique
Example: Merge Sort
Dynamic Programming
In this the problem is solved in following steps:
1. Defining structure of optimal solution
2. Defines value of optimal solutions repeatedly.
3. Obtaining values of optimal solution in bottom-up fashion
4. Getting final optimal solution from obtained values
Non-Recursive
Bottom Up Technique
Example: Strassen's Matrix Multiplication
Divide and Conquer:
This paradigm involves three stages:
Divide the problem into smaller sub-problems
Conquer, i.e., solve these smaller sub-problems
Combine these sub-problems' solutions to get the final answer.
Dynamic Programming:
DP is an optimization of recursive solutions. The primary difference it makes is that it stores the solution to sub-problems, which can later be accessed during the process of finding solutions of the remaining sub-problems. This is done so that we don't have to calculate the solution to a sub-problem every time, rather we can simply look it up the computer memory to retrieve its value, given that it has been solved earlier. We can simply add this as our base case in recursion. For example, we are solving a problem through recursion, we can store the solutions to sub-problems in an array and access them by adding the relevant code in one of our base cases in the recursive method.
There are two ways in which DP is done:
Consider a problem: To find factorial of x.
Tabulation: We use the bottom up approach, that is we go from the smallest numbers all the way upto x, to find the solution.
Pseudo Code:
1. int array
2. for int=1, i<=x, i++
3. array[i] = array[i-1]*i
Memoization: We use the top down approach, that is we take the problem and then break it down into smaller parts and solve them, to get the final solution
Pseudo Code:
fac():
1. int array
2. if(x==0): return 1
3. if(array[x]!=null): return array[x]
4. return array[x] = x*fac(x-1)