What's the difference between backtracking and depth first search?
Backtracking is a more general purpose algorithm.
Depth-First search is a specific form of backtracking related to searching tree structures. From Wikipedia:
One starts at the root (selecting some node as the root in the graph case) and explores as far as possible along each branch before backtracking.
It uses backtracking as part of its means of working with a tree, but is limited to a tree structure.
Backtracking, though, can be used on any type of structure where portions of the domain can be eliminated - whether or not it is a logical tree. The Wiki example uses a chessboard and a specific problem - you can look at a specific move, and eliminate it, then backtrack to the next possible move, eliminate it, etc.
I think this answer to another related question offers more insights.
For me, the difference between backtracking and DFS is that backtracking handles an implicit tree and DFS deals with an explicit one. This seems trivial, but it means a lot. When the search space of a problem is visited by backtracking, the implicit tree gets traversed and pruned in the middle of it. Yet for DFS, the tree/graph it deals with is explicitly constructed and unacceptable cases have already been thrown, i.e. pruned, away before any search is done.
So, backtracking is DFS for implicit tree, while DFS is backtracking without pruning.
IMHO, most of the answers are either largely imprecise and/or without any reference to verify. So let me share a very clear explanation with a reference.
First, DFS is a general graph traversal (and search) algorithm. So it can be applied to any graph (or even forest). Tree is a special kind of Graph, so DFS works for tree as well. In essence, let’s stop saying it works only for a tree, or the likes.
Based on [1], Backtracking is a special kind of DFS used mainly for space (memory) saving. The distinction that I’m about to mention may seem confusing since in Graph algorithms of such kind we’re so used to having adjacency list representations and using iterative pattern for visiting all immediate neighbors (for tree it is the immediate children) of a node, we often ignore that a bad implementation of get_all_immediate_neighbors may cause a difference in memory uses of the underlying algorithm.
Further, if a graph node has branching factor b, and diameter h (for a tree this is the tree height), if we store all immediate neighbors at each steps of visiting a node, memory requirements will be big-O(bh). However, if we take only a single (immediate) neighbor at a time and expand it, then the memory complexity reduces to big-O(h). While the former kind of implementation is termed as DFS, the latter kind is called Backtracking.
Now you see, if you’re working with high level languages, most likely you’re actually using Backtracking in the guise of DFS. Moreover, keeping track of visited nodes for a very large problem set could be really memory intensive; calling for a careful design of get_all_immediate_neighbors (or algorithms that can handle revisiting a node without getting into infinite loops).
[1] Stuart Russell and Peter Norvig, Artificial Intelligence: A Modern Approach, 3rd Ed
According to Donald Knuth, it's the same.
Here is the link on his paper about Dancing Links algorithm, which is used to solve such "non-tree" problems as N-queens and Sudoku solver.
Backtracking, also called depth-first search
Backtracking is usually implemented as DFS plus search pruning. You traverse search space tree depth-first constructing partial solutions along the way. Brute-force DFS can construct all search outcomes, even the ones, that do not make sense practically. This can be also very inefficient to construct all solutions (n! or 2^n). So in reality as you do DFS, you need to also prune partial solutions, which do not make sense in context of the actual task, and focus on the partial solutions, which can lead to valid optimal solutions. This is the actual backtracking technique - you discard partial solutions as early as possible, make a step back and try to find local optimum again.
Nothing stops to traverse search space tree using BFS and execute backtracking strategy along the way, but it doesn't make sense in practice because you would need to store search state layer by layer in the queue, and tree width grows exponentially to the height, so we would waste a lot of space very quickly. That's why trees are usually traversed using DFS. In this case search state is stored in the stack (call stack or explicit structure) and it can't exceed tree height.
Usually, a depth-first-search is a way of iterating through an actual graph/tree structure looking for a value, whereas backtracking is iterating through a problem space looking for a solution. Backtracking is a more general algorithm that doesn't necessarily even relate to trees.
I would say, DFS is the special form of backtracking; backtracking is the general form of DFS.
If we extend DFS to general problems, we can call it backtracking.
If we use backtracking to tree/graph related problems, we can call it DFS.
They carry the same idea in algorithmic aspect.
DFS describes the way in which you want to explore or traverse a graph. It focuses on the concept of going as deep as possible given the choice.
Backtracking, while usually implemented via DFS, focuses more on the concept of pruning unpromising search subspaces as early as possible.
IMO, on any specific node of the backtracking, you try to depth firstly branching into each of its children, but before you branch into any of the child node, you need to "wipe out" previous child's state(this step is equivalent to back walk to the parent node). In other words, each siblings state should not impact each other. On the contrary, during normal DFS algorithm, you don't usually have this constraint, you don't need to wipe out(back track) previous siblings state in order to construct next sibling node.
Depth first is an algorithm for traversing or searching a tree. See here. Backtracking is a much more broad term that is used whereever a solution candidate is formed and later discarded by backtracking to a former state. See here. Depth first search uses backtracking to search a branch first (solution candidate) and if not successful search the other branch(es).
Depth first search(DFS) and backtracking are different searching and traversing algorithms. DFS is more broad and is used in both graph and tree data structure while DFS is limited to tree. With that being said,it does not mean DFS can't be used in graph. It is used in graph as well but only produce spanning tree, a tree without loop(multiple edge starting and ending at same vertex). That is why it is limited to tree.
Now coming back to backtracking, DFS use backtracking algorithm in tree data structure so in tree, DFS and backtracking are similar.
Thus,we can say, they are in same in tree data structure whereas in graph data structure, they are not same.
Idea - Start from any point, check if its the desired endpoint, if yes then we found a solution else goes to all next possible positions and if we can't go further then return to the previous position and look for other alternatives marking that current path wont lead us to the solution.
Now backtracking and DFS are 2 different names given to the same idea applied on 2 different abstract data types.
If the idea is applied on matrix data structure we call it backtracking.
If the same idea is applied on tree or graph then we call it DFS.
The cliche here is that a matrix could be converted to a graph and a graph could be transformed to a matrix. So, we actually apply the idea. If on a graph then we call it DFS and if on a matrix then we call it backtracking.
The idea in both of the algorithm is same.
The way I look at DFS vs. Backtracking is that backtracking is much more powerful. DFS helps me answer whether a node exists in a tree, while backtracking can help me answer all paths between 2 nodes.
Note the difference: DFS visits a node and marks it as visited since we are primarily searching, so seeing things once is sufficient. Backtracking visits a node multiple times as it's a course correction, hence the name backtracking.
Most backtracking problems involve:
def dfs(node, visited):
visited.add(node)
for child in node.children:
dfs(child, visited)
visited.remove(node) # this is the key difference that enables course correction and makes your dfs a backtracking recursion.
In a depth-first search, you start at the root of the tree and then explore as far along each branch, then you backtrack to each subsequent parent node and traverse it's children
Backtracking is a generalised term for starting at the end of a goal, and incrementally moving backwards, gradually building a solution.
Backtracking is just depth first search with specific termination conditions.
Consider walking through a maze where for each step you make a decision, that decision is a call to the call stack (which conducts your depth first search)... if you reach the end, you can return your path. However, if you reach a deadend, you want to return out of a certain decision, in essence returning out of a function on your call stack.
So when I think of backtracking I care about
State
Decisions
Base Cases (Termination Conditions)
I explain it in my video on backtracking here.
An analysis of backtracking code is below. In this backtracking code I want all of the combinations that will result in a certain sum or target. Therefore, I have 3 decisions which make calls to my call stack, at each decision I either can pick a number as part of my path to reach the target num, skip that number, or pick it and pick it again. And then if I reach a termination condition, my backtracking step is just to return. Returning is the backtracking step because it gets out of that call on the call stack.
class Solution:
"""
Approach: Backtracking
State
-candidates
-index
-target
Decisions
-pick one --> call func changing state: index + 1, target - candidates[index], path + [candidates[index]]
-pick one again --> call func changing state: index, target - candidates[index], path + [candidates[index]]
-skip one --> call func changing state: index + 1, target, path
Base Cases (Termination Conditions)
-if target == 0 and path not in ret
append path to ret
-if target < 0:
return # backtrack
"""
def combinationSum(self, candidates: List[int], target: int) -> List[List[int]]:
"""
#desc find all unique combos summing to target
#args
#arg1 candidates, list of ints
#arg2 target, an int
#ret ret, list of lists
"""
if not candidates or min(candidates) > target: return []
ret = []
self.dfs(candidates, 0, target, [], ret)
return ret
def dfs(self, nums, index, target, path, ret):
if target == 0 and path not in ret:
ret.append(path)
return #backtracking
elif target < 0 or index >= len(nums):
return #backtracking
# for i in range(index, len(nums)):
# self.dfs(nums, i, target-nums[i], path+[nums[i]], ret)
pick_one = self.dfs(nums, index + 1, target - nums[index], path + [nums[index]], ret)
pick_one_again = self.dfs(nums, index, target - nums[index], path + [nums[index]], ret)
skip_one = self.dfs(nums, index + 1, target, path, ret)
In my opinion, the difference is pruning of tree. Backtracking can stop (finish) searching certain branch in the middle by checking the given conditions (if the condition is not met). However, in DFS, you have to reach to the leaf node of the branch to figure out if the condition is met or not, so you cannot stop searching certain branch until you reach to its leaf nodes.
The difference is: Backtracking is a concept of how an algorithm works, DFS (depth first search) is an actual algorithm that bases on backtracking. DFS essentially is backtracking (it is searching a tree using backtracking) but not every algorithm based on backtracking is DFS.
To add a comparison: Backtracking is a concept like divide and conqueror. QuickSort would be an algorithm based on the concept of divide and conqueror.
Related
I am currently taking an Artificial Intelligence course and learning about DFS and BFS
if we take the following example:
From my understanding, the BFS algorithm will explore the first level containing then the second level containing D,E,F,G , etc... till it reaches the last level
I am lost concerning which node between B and C (for example) will the BFS expand first? Originally I thought it is different every time and by convention, we choose to do illustrate it done from left to right (so exploring B then C) but my professor said that our choice between B and C depends on each case and we choose the "shallowest node first", in made examples there isn't a distance factor between A,B and A,C so how could one choose then?
My question is the same concerning DFS where I was told to choose the "deepest node first", I am aware that there are pre-order versions and others but the book "Artificial Intelligence - A Modern Approach, by Stuart Russel" didn't get into them
I tried checking the CLRE algorithms book for more help but the expansion is done based on the order in the adjacency list which didn't really help.
In the definition, BFS doesn't state any rule regarding which node you should visit first as long as it is in the same level. However, depending on your actual implementation, there will be some order. For example for a binary tree (like in the example you have provided), the implementation of BFS might prefer going left then right, other implementations might do the opposite.
Conclusion: it doesn't really matter, as long as we are considering the general definition of BFS / DFS. If you find the order of visiting nodes in the same level specified in some book, that's not BFS, that's a modified BFS, meaning a specific implementation / variant.
Actually there are many variants for DFS / BFS (like right-first DFS on grids) but as I said, all of them are specific implementations, the general definition doesn't state the order of nodes in the same level.
Here is another view of the problem:
A BFS is usually implemented by using a queue, a first-in first-out data structure, while
A DFS is usually implemented by using a stack, a first-in last-out data structure.
Therefore, how to choose between nodes with the "same priority" depends on how you put those nodes in the stack/queue. There is no fixed rule about it.
I am trying to prove the following algorithm to see if a there exists a path from u to v in a graph G = (V,E).
I know that to finish up the proof, I need to prove termination, the invariants, and correctness but I have no idea how. I think I need to use induction on the while loop but I am not exactly sure how.
How do I prove those three characteristics about an algorithm?
Disclaimer: I don't know how much formal you want your proof to be and I'm not familiar with formal proofs.
induction on the while loop: Is it true at the beginning? Does it remain true after a step (quite simple path property)?
same idea, induction on k (why k+1???): Is it true at the beginning? Does it remain true after a step (quite simple path property)?
Think Reach as a strictly increasing set.
Termination: maybe you can use a quite simple property linked to the diameter of the graph?
(This question could probably be better answered elsewhere, on https://cstheory.stackexchange.com/ maybe?)
There is a lot of possibilities. For example, for a Breadth First Search, we note that:
(1) The algorithm never visits the same node twice.(as any path back must be >= the length that put it in the discovered pile already.
(2) At every step, it adds exactly one node.
Thus, it clearly must terminate on any finite graph, as the set of nodes which are discoverable cannot be larger than the set of nodes which are in the graph.
Finally, since, give a start node, it will only terminate when it has reached every node which is connected by any path to the start node, it will always find a path between the start and target if it exists.
You can rewrite these logical steps above in deeper rigour if you like, for example, by showing that the list of visited nodes is strictly increasing, and non convergent (i.e. adding one to something repeatedly tends to infinity) and the termination condition must be met at at most some finite value, and a non convergent increasing function always intersects a given bound exactly once.
BFS is an easy example because it has such simple logic, but proving these things for a given algorithm may be extremely tricky.
The idea of deleting a node in BST is:
If the node has no child, delete it and update the parent's pointer to this node as null
If the node has one child, replace the node with its children by updating the node's parent's pointer to its child
If the node has two children, find the predecessor of the node and replace it with its predecessor, also update the predecessor's parent's pointer by pointing it to its only child (which only can be a left child)
the last case can also be done with use of a successor instead of predecessor!
It's said that if we use predecessor in some cases and successor in some other cases (giving them equal priority) we can have better empirical performance ,
Now the question is , how is it done ? based on what strategy? and how does it affect the performance ? (I guess by performance they mean time complexity)
What I think is that we have to choose predecessor or successor to have a more balanced tree ! but I don't know how to choose which one to use !
One solution is to randomly choose one of them (fair randomness) but isn't better to have the strategy based on the tree structure ? but the question is WHEN to choose WHICH ?
The thing is that is fundamental problem - to find correct removal algorithm for BST. For 50 years people were trying to solve it (just like in-place merge) and they didn't find anything better then just usual algorithm (with predecessor/successor removing). So, what is wrong with classic algorithm? Actually, this removing unbalances the tree. After several random operations add/remove you'll get unbalanced tree with height sqrt(n). And it is no matter what you choosed - remove successor or predecessor (or random chose beetwen these ways) - the result is the same.
So, what to choose? I'm guessing random based (succ or pred) deletion will postpone unbalancing of your tree. But, if you want to have perfectly balanced tree - you have to use red-black ones or something like that.
As you said, it's a question of balance, so in general the method that disturbs the balance the least is preferable. You can hold some metrics to measure the level of balance (e.g., difference from maximal and minimal leaf height, average height etc.), but I'm not sure whether the overhead worth it. Also, there are self-balancing data structures (red-black, AVL trees etc.) that mitigate this problem by rebalancing after each deletion. If you want to use the basic BST, I suppose the best strategy without apriori knowledge of tree structure and the deletion sequence would be to toggle between the 2 methods for each deletion.
Is the greedy best-first search algorithm different from the best-first search algorithm?
The wiki page has a separate paragraph about Greedy BFS but it's a little unclear.
My understanding is that Greedy BFS is just BFS where the "best node from OPEN" in wikipedia's algorithm is a heuristic function one calculates for a node. So implementing this:
OPEN = [initial state]
CLOSED = []
while OPEN is not empty
do
1. Remove the best node from OPEN, call it n, add it to CLOSED.
2. If n is the goal state, backtrace path to n (through recorded parents) and return path.
3. Create n's successors.
4. For each successor do:
a. If it is not in CLOSED: evaluate it, add it to OPEN, and record its parent.
b. Otherwise: change recorded parent if this new path is better than previous one.
done
with "best node from OPEN" being a heuristic function estimating how close the node is to the goal, is actually Greedy BFS. Am I right?
EDIT: Comment on Anonymouse's answer:
So essentially a greedy BFS doesn't need an "OPEN list" and should base its decisions only on the current node? Is this algorithm GBFS:
1. Set START as CURRENT node
2. Add CURRENT to Path [and optinally, to CLOSED?]
3. If CURRENT is GOAL, exit
4. Evaluate CURRENT's successors
5. Set BEST successor as CURRENT and go to 2.
"Best first" could allow revising the decision, whereas, in a greedy algorithm, the decisions should be final, and not revised.
For example, A*-search is a best-first-search, but it is not greedy.
However, note that these terms are not always used with the same definitions. "Greedy" usually means that the decision is never revised, eventually accepting suboptimal solutions at the benefit of improvements in running time. However, I bet you will find situations where "greedy" is used for the combination of "best first + depth first" as in "try to expand the best next step until we hit a dead end, then return to the previous step and continue with the next best there" (which I would call a "prioritized depth first").
Also, it depends on which level of abstraction you are talking about. A* is not greedy in "building a path". It's fine with keeping a large set of open paths around. It is however greedy in "expanding the search space" towards the true shortest path.
BFS is an instance of tree search and graph search algorithms in which a node is selected for expansion based on the evaluation function f(n) = g(n) + h(n), where g(n) is length of the path from the root to n and h(n) is an estimate of the length of the path from n to the goal node. In a BFS algorithm, the node with the lowest evaluation (i.e. lowest f(n)) is selected for expansion.
Greedy BFS uses the following evaluation function f(n) = h(n), which is just the heuristic function h(n), which estimates the closeness of n to the goal. Hence, greedy BFS tries to expand the node that is thought to be closest to the goal, without taking into account previously gathered knowledge (i.e. g(n)).
To summarize, the main difference between these (similar) search methods is the evaluation function.
As a side note, the A* algorithm is a best-first search algorithm in which the heuristic function h is an admissible heuristic (i.e. h is always an underestimation of the perfect heuristic function h*, for all n). A* is not a gredy BFS algorithm because its evaluation function is f(n) = g(n) + h(n).
Ten years after I asked this question, I got back to it and finally understood what that article in Wikipedia was saying.
Greedy BFS is greedy in expanding a potentially better successor of the current node. The difference between the two algorithms is in the loop that handles the evaluation of successors. Best-first search always exhausts the current node's successors by evaluating them and continues with the best one from them:
4. For each successor do:
a. If it is not in CLOSED: evaluate it, add it to OPEN, and record its parent.
b. Otherwise: change recorded parent if this new path is better than previous one.
Greedy BFS doesn't expand all successors of a node if it finds one that has a better heuristic than the current node. Instead it greedily expands this potentially better node, leaving some of the current node's successors unexpanded. This means the current node shouldn't be removed from the OPEN list unless all its successors have been evaluated. This is the pseudo-code:
OPEN = [initial state]
CLOSED = []
while OPEN is not empty
do
1. Remove the best node from OPEN, call it n, add it to CLOSED.
2. If n is the goal state, backtrace path to n (through recorded parents) and return path.
3. For each successor do:
a. If it is not in CLOSED:
i. Evaluate it, add it to OPEN, and record its parent
ii. If it has a better heuristic than n: remove n from CLOSED, add n to OPEN
(after the current successor) and break from loop 3.
b. Otherwise: change recorded parent if this new path is better than previous one.
done
I have removed the step 3. Create n's successors. from the BFS code above because some of n's successors may not be evaluated, so there is no use creating them. Instead each successor should be created and immediately evaluated in 3. For each successor do:.
As far as I understand, "best-first search" is only a collective name of a particular search technique in which you use a heuristic evaluation function h(n).
So, A* and greedy best-first search are algorithms that fall into this category.
I’m trying to build a parallel implementation of a min-max search. My current approach is to materialize the tree to a small depth and then do the normal thing from each of these nodes.
The simple way to do this is to compute the heuristic value for each leaf and then sweep up and compute the min/max. The problem is that it this omits alpha/beta pruning at the upper levels and makes for a major performance hit.
My first “solution” to that was to push the min/max up after each leaf is computed. This gives updating value so I can scan up the tree and check if a leaf should be pruned.
The problem is that it's totally broken. (2 days of debugging to notice that, darn I feel stupid)
Now for the question:
Is there a way to build a min-max tree that allows the leafs to be evaluated in random order and also allows alpha/beta pruning?
Check out parallel game tree search, e.g. this paper.
I think I have found a solution but I don't like it in a few regards:
Annotate the tree with the number of unfinished children.
After a leaf is evaluated, update it's parent, decrement the parent's count
If that count just reached zero, update it's parent, decrement that count
Lather, rise, repeat
Alpha/beta pruning works as expected.
The problems with this is that with random order evaluation, a lot more nodes will get evaluated before stuff starts getting pruned. On the other hand, that might be mitigated by better ordering of the leafs.