I want to parallelize my sudoku solver program with MPI. The current serial code relies on backtracking with depth-first search. I did some research, but I am still not sure how to do it.
Some say that program must do a breadth-first search to get some data in master process and then use slave processes with this data. So that the slave processes will do depth-first search using this data.
Also I saw that some depth-first search parallelization examples use work sharing or work stealing methods. But in case of sudoku, I am not sure that using this technique can handle process relations, work queue and process size because of sudoku's solving methodology.
Any ideas?
Thank you.
This is not an answer pertaining to Sudoku, more to the fact that you specify your serial algorithm uses a depth-first search. Depth-first search is a problem that is known to be difficult to parallelize, despite not appearing to be 'inherently serial'.
There are parallel DFS implementations however. For example this 1987 paper presents a parallel DFS algorithm. The general principle is that each processor searches a different set of paths until it reaches a leaf (or an arbitrary search depth), and when it has completed it's subset of paths, chooses a new unexplored branch.
If you are keen to implement a parallel DFS I would recommend reading that article and having a go implementing that algorithm. However I think there are probably more intelligent parallel sudoku algorithms that do not use DFS, for example it could be solved using Constraint Propagation.
Related
Both NN and Greedy Search algorithms have a Greed nature, and both have tendency towards the lowest cost/distance (my understanding may be incorrect though). But what makes them different in a way that each one can be classified into a distinct algorithm group is somehow unclear to me.
For instance, if I can solve a particular problem using NN, I can surely solve it with Greedy Search algorithm as well specially if minimization is the case. I came to this conclusion because when I start coding them I come across very similar implementations in code although the general concept behind both might be different. Sometimes I can't even tell if the implementation follows NN or Greedy Search.
I have done my homework well and searched enough on Google, but couldn't find a decent explanation on what distinguishes them from one another. Any such explanation is indeed appreciated.
Hmm, at a very high level they both driven by heuristics in order to evaluate a given solution against an ideal solution. But, whilst a greedy search algo outputs a solution for a given input, the NN trains a model that will generate solutions for given inputs. So at a very very high level, you can think that the NN generates a solution finder, whereas the greedy search is a harcoded solution finder.
In other words, the NN will generate "code" (i.e. the model (aka the weights)) that finds solutions to the problem when provided to the same network topology. The greedy search is you actually writing the code that finds the solution to the problem. This is quite wishy washy though, I'm sure there is a much more concise, academically sound way of expressing what I've just said
All of what I've just said in based on the assumption that by "Greedy search" you meant the algorithms to solve problems such as travelling sales man.
Another way to think of it is:
In greedy search, you write an algorithm that solves a search problem (find me the graph that best describes the relationship, based on provided heuristic(s), between data point A and data point B).
When you write a neural network, you declare a network topology, provide some initially "random" weights and some heuristics to measure output errors and then train the networks weights via a plethora of different methods (back prop, GAN etc). These weights can then be used as a solver for novel problems.
For what it's worth, I don't think an NN would be a good approach to generate a solver for travelling sales man problem. You would be far better off just using a common graph search algorithm..
You might have to read this twice so that my idea becomes clear. Please be patient.
I'm looking for existing work for exhausive search for algorithms for given problems. Exhaustive search is also known as brute force search, or simply brute force.
Other exhaustive search algorithms search for a solution for a given problem. Usually, a solution for such a problem is some data which fulfills some requirements.
Exhaustive search example:
You want the solution for a KNAPSACK problem. That is the objects which can be packed into the bag so that there is no other combination of objects which fit into the bag and which would sum up to a bigger value than your result combination.
You can solve this by going over all possible combinations (exhaustive) and search for the one which fits into the bag and which is the most valuable one of those combinations.
What I'm looking for is just a special case of exhaustive search: Exhaustive search which searches for an algorithm as a solution. So in the end, I'm looking for an algorithm which searches for an algorithm which solves some given problem.
You might say: Go google it. Well, yes I've done that. The difficulty I'm facing here is that googling for "algorithm which searches for another algorithm" results in all the same as "another search algorithm". Obviously, this has too many unwanted results, so I'm stuck here.
How can I find existing work related to exhaustive search for algorithms?
More specifically: Has there any software been written for this? Can you point me to any links or algorithm names/better keywords in relation to this topic?
Update:
The purpose for which I'm looking for such an algorithm search is solving problems for which no good heuristics are known, e.g. prooving-algorithms or trying to finding other solution algorithms for problems which might or might not be NP-complete problems (thus proving that the problem is not NP-complete if a faster algorithm can be found; without any human interaction).
You seem to be looking for "program synthesis", which can work in some limited instances, provided you can correctly and formally specify what your algorithm is supposed to do (without giving an implementation).
Synthesis is an effective approach to build gate-level circuits, but applying synthesis to software is so far still more of a research avenue than practical.
Still, here are a couple of references on the subject,
(Some of the most advanced work in the area in my opinion, has a tool)
program sketching by Armando Solar-Lezama
Check out Microsoft research page on the topic, they think it's hot topic: http://research.microsoft.com/en-us/um/people/sumitg/pubs/synthesis.html
Some other similar stuff I've seen :
Model Checking-Based Genetic Programming with an Application to Mutual Exclusion. (Katz & Peled # TACAS '08), they have a more recent version on ArXiv : http://arxiv.org/abs/1402.6785
Essentially the search space is explored (exhaustively) using a model-checker.
I wouldn't think it would be possible (at least without some restriction on the class of algorithms) and, in any event, the search space would be so large that it would make ordinary brute-force tame by comparison. You could, for example, enumerate algorithms for a specific model of computation (e.g. Turing machines) but then the Halting problem rears its head -- how can you tell if it solves your problem? If you have a family of algorithms which depend on discrete parameters then you could of course brute force the choice of the parameters.
There is an extensive literature on Genetic Programming (see https://en.wikipedia.org/wiki/Genetic_programming ). That might give you something to go on. In particular -- the tree data structures that are often used in this context (essentially expression trees or more generally abstract syntax trees) could admit a brute force enumeration.
Take a look at Neural Turing Machines by Alex Graves, Greg Wayne and Ivo Danihelka from Google DeepMind.
Abstract:
We extend the capabilities of neural networks by coupling them to
external memory resources, which they can interact with by attentional
processes. The combined system is analogous to a Turing Machine or Von
Neumann architecture but is differentiable end-to-end, allowing it to
be efficiently trained with gradient descent. Preliminary results
demonstrate that Neural Turing Machines can infer simple algorithms
such as copying, sorting, and associative recall from input and output
examples.
This paper, entitled "Systematic search for lambda expressions" performs exhaustive enumeration in the space of small type-safe functional programs expressed in the lambda calculus.
I used Genetic Programming for evolving/generating heuristics for the Travelling Salesman Problem. The evolved heuristic is better than the Nearest Neighbor on the tested problems (random graphs and other graphs taken from TSPLIB). If you want the source code, you can download it from here: http://www.tcreate.org/evolve_heuristics.html
Does anyone knows if there exists a breadth first(from multiple sources) implementation in either of the graph processing systems- Giraph, Pregel or Graphchi.
Or please tell some easier implementation on either of the systems.
In the Giraph user mail list, one can find a few discussions - and i guess an implementation also - of BFS implementation.
I have made this type of search for Giraph in the past and they are available at:
https://github.com/MarcoLotz/GiraphBFSSO
https://github.com/MarcoLotz/GiraphBFSTO
The difference between them is that one is target oriented and other one is structure oriented.
Although they are not from multiple start vertex, one can easily modify the code to support it :)
You are looking for the multi-seed breadth first search (BFS) algorithm.
For Giraph, this is still an open issue as can be read in this
feature request.
For Pregel, you can not expect to find any open algorithm as Pregel is a closed-source graph system at Google.
I guess the easiest thing would be to use the code from Github and execute it for each source separately. One idea to optimize the algorithm's runtime complexity is to execute BFS for the first seed vertex and reuse your results for subsequent vertex seeds (the first BFS results in a spanning tree which can be easily transformed to a BFS order for any given seed vertex).
Nevertheless, KISS suggests to simply execute k times BFS for k seed vertices until you run into performance issues (which is unlikely due to linear runtime complexity of BFS).
I have implemented an A* search algorithm for finding a shortest path between two states.
Algorithm uses a hash-map for storing best known distances for visited states. And one hash-map for storing child-parent relationships needed for reconstruction of the shortest path.
Here is the code. Implementation of the algorithm is generic (states only need to be "hashable" and "comparable") but in this particular case states are pairs (vectors) of ints [x y] and they represent one cell in a given heightmap (cost for jumping to neighboring cell depends on the difference in heights).
Question is whether it's possible to improve performance and how? Maybe by using some features from 1.2 or future versions, by changing logic of the algorithm implementation (e.g. using different way to store path) or changing state representation in this particular case?
Java implementation runs in an instant for this map and Clojure implementation takes about 40 seconds. Of course, there are some natural and obvious reasons for this: dynamic typing, persistent data structures, unnecessary (un)boxing of primitive types...
Using transients didn't make much difference.
Using priority-map instead of sorted-set
I first used sorted-set for storing open nodes (search frontier), switching to priority-map improved performance: now it takes 15-20 seconds for this map (before it took 40s).
This blog post was very helpful. And "my" new implementation is pretty much the same.
New a*-search can be found here.
I don't know Clojure, but I can give you some general advice about improving the performance of Vanilla A*.
Consider implementing IDA*, which is a variant of A* that uses less memory, if it's suitable for your domain.
Try a different heuristic. A good heuristic can have a significant impact on the number of node expansions required.
Use a cache, Often called a "transposition table" in search algorithms. Since search graphs are usually Directed Acyclic Graphs and not true trees, you can end up repeating the search of a state more than once; a cache to remember previously-searched nodes reduces node expansions.
Dr. Jonathan Schaeffer has some slides on this subject:
http://webdocs.cs.ualberta.ca/~jonathan/Courses/657/Notes/10.Single-agentSearch.pdf
http://webdocs.cs.ualberta.ca/~jonathan/Courses/657/Notes/11.Evaluations.pdf
I am currently in the process of working on an scheduling application and have run into a small snag. Because the field is heavily regulated for safety reasons the software is required to check a number of interdependent conditions to ensure a trip is possible. Instead of a nice tree of conditions - which would be simple to implement - I have this hideous directed graph. Now for the most part this wouldn't be difficult at all except for the fact I may not know all the information required in advance but still need to perform as much of the validation as possible.
I could implement this as a rat's nest of if/else statements but that would be a maintainability nightmare since the regulations change on a fairly regular basis. Since there are no cycles in the graph I'm thinking that some form of breadth-first approach is probably optimal. Am I on the right track? Are there any other alternative techniques for performing this kind of task?
The solution depends completely on what the directed acyclic graph (DAG) you speak of actually represents. Are the nodes AND and OR nodes, or are they conditional braches?
If they are conditional branches you don't need any breadth-first search because there is nothing to search for; you just take the branches according to the evaluated conditions. Yes, it could be easily implemented as a GOTO spaghetti. Another option is to create a database of the nodes (or a datastructure) and have a "virtual machine" which executes through the nodes one by one. This makes it more maintainable but also slower.
If it is an AND/OR/NOT graph then you need to evaluate the truth values for the nodes starting from the leaves. This is not breadth-first search, but kind-of reverse breadth first approach. You calculate the values for the leaves and then calculate the values of the internal nodes backwards; eventually you get an evaluation of the root node (true / false).
Sounds like you are trying to solve a common compilation problem called '[constant folding][1]'. (http://en.wikipedia.org/wiki/Constant_folding).
The good new is that it applies to DAGs (direct acyclic graph) and not just to trees. Basically the idea is to compute what you can from partial expressions. Things as simple as True AND X = X, or True OR X = True help pruning large parts of the tree. (The trivial implementation I know of is a matter of depth-first and backtracking more than breadth-first, but anyway it's not much code).
I still wonder why you got a graph an not an expression tree. If you can compute A from B or B from A you usually do not have both A and B as input at the same time. Then it should be possible to express the problem like a set of expression trees depending on available inputs. But it's raw guess. Can you give more details (exemple) on why you have this graph ?