For finding the best solution in the assignment problem it's easy to use the Hungarian Algorithm.
For example:
A | 3 4 2
B | 8 9 1
C | 7 9 5
When using the Hungarian Algorithm on this you become:
A | 0 0 1
B | 5 5 0
C | 0 1 0
Which means A gets assigned to 'job' 2, B to job 3 and C to job 1.
However, I want to find the second best solution, meaning I want the best solution with a cost strictly greater that the cost of the optimal solution. According to me I just need to find the assignment with the minimal sum in the last matrix without it being the same as the optimal. I could do this by just searching in a tree (with pruning) but I'm worried about the complexity (being O(n!)). Is there any efficient method for this I don't know about?
I was thinking about a search in which I sort the rows first and then greedily choose the lowest cost first assuming most of the lowest costs will make up for the minimal sum + pruning. But assuming the Hungarian Algorithm can produce a matrix with a lot of zero's, the complexity is terrible again...
What you describe is a special case of the K best assignments problem -- there was in fact a solution to this problem proposed by Katta G. Murty in the following 1968 paper "An Algorithm for Ranking all the Assignments in Order of Increasing Cost." Operations Research 16(3):682-687.
Looks like there are actually a reasonable number of implementations of this, at least in Java and Matlab, available on the web (see e.g. here.)
In r there is now an implementation of Murty's algorithm in the muRty package.
CRAN
GitHub
It covers:
Optimization in both minimum and maximum direction;
output by rank (similar to dense rank in SQL), and
the use of either Hungarian algorithm (as implemented in clue) or linear programming (as implemented in lpSolve) for solving the initial assignment(s).
Disclaimer: I'm the author of the package.
Related
In today's edition of the Guardian (a newspaper in the UK), in the "Pyrgic puzzles" section on page 43 by Chris Maslanka, the following puzzle was given:
The 3 wise men ... went to Herrods to do their Christmas shopping. Caspar bought Gold, Melchior bought Frankincense, and Balthazar bought a copy of the Daily Myrrh. The cashier tapped in the number of euros of each of these things cost, and meant to add the three numbers, but multiplied them instead. ... the marvel of the thing was that the result was exactly the same: €65.52. What were the three sums [I assume he meant the three numbers]?
My interpretation is: Find an a, b and c such that a + b + c = abc = 65.52 (exactly) where a, b and c are positive decimal numbers with no more than two decimal places. It follows that a, b and c must also be less than 65.52 (approximately).
My approach is thus: I shall find all the candidate sets of a, b and c where a + b + c = 6552 and a, b and c are integers from {1 ... 6550} (Notionally I have multiplied all the operands by 100 for convenience). Then, for all the candidate sets, it is trivial to satisfy the other condition by dividing all the operands by 100 then multiplying them (with arbitrary-precision arithmetic).
This, as I see it, is an instance of the subset sum problem. So I implemented a dirty (exponential time) algorithm which found one distinct solution: a=0.52, b=2, c=63.
Ok, there are better algorithms for the subset sum problem, but don't you think this is getting a little out-of-reach for an average Guardian reader?
On page 40 the answer is listed:
This is easy, by trial and error. Guess 52p for the Daily Myrrh. But by multiplying by 0.52 is roughly halving, so we need one sum to be about 2; so try 2 X 63 X 0.52. Et voilà. Is this answer unique?
Well, we know that the answer is unique (disregarding the other permutations of 2, 63 and 0.52).
What I want to know is: How can this be "easy"? Am I right in characterising the puzzle as an instance of the subset sum problem? Have I overlooked some characteristic of the puzzle which can be utilized to simplify the solution? Was anyone able to adopt a similar "trial and error" approach and if so can they take me through it? Is Chris Maslanka simply undaunted by NP-complete problems?
No, it is not an instance of the subset sum problem, because:
The subset size is limited to 3, making it O(n^3) solution worst case with naive exhaustive search (and not exponential)
There is additional data in here, the product of the numbers.
You are not actually given a set, a set of all integers is just a subproblem of subset-sum, a much easier one.
The important thing to understand here is: if a problem can be solved by an NP-Hard problem - it doesn't mean it is NP-Hard as well, the other way around holds - if you have a problem, and you can solve some NP-Hard problem (polynomially) with it, then your problem is NP-Hard. It is called polynomial reduction1.
The approach is easy because all you have to do is "guess" (by iterating all candidates) a value for a, and from this you can derive what is the possible solution for b,c - (2 variables, two equations if a is known - and in each iteration - it is), thus the solution is even linear - not only sub exponential.
It might even be optimized to use a variation of binary search to get a sub-linear optimization, but I cannot think of that optimization at the moment.
(1) Note: this is some intuitive explanation, and not a formal definition.
I have a simple algorithmic question. I would be grateful if you could help me.
We have some 2 dimensional points. A positive weight is associated to them (a sample problem is attached). We want to select a subset of them which maximizes the weights and neither of two selected points overlap each other (for example, in the attached file, we cannot select both A and C because they are in the same row, and in the same way we cannot select both A and B, because they are in the same column.) If there is any greedy (or dynamic) approach I can use. I'm aware of non-overlapping interval selection algorithm, but I cannot use it here, because my problem is 2 dimensional.
Any reference or note is appreciated.
Regards
Attachment:
A simple sample of the problem:
A (30$) -------- B (10$)
|
|
|
|
C (8$)
If you are OK with a good solution, and do not demand the best solution - you can use heuristical algorithms to solve this.
Let S be the set of points, and w(s) - the weightening function.
Create a weight function W:2^S->R (from the subsets of S to real numbers):
W(U) = - INFINITY is the solution is not feasible
Sigma(w(u)) for each u in U otherwise
Also create a function next:2^S -> 2^2^S (a function that gets a subset of S, and returns a set of subsets of S)
next(U) = V you can get V from U by adding/removing one element to/from U
Now, given that data - you can invoke any optimization algorithm in the Artificial Intelligence book, such as Genetic Algorithm or Hill Climbing.
For example, Hill Climbing with random restarts, will be something like that:
1. best<- -INFINITY
2. while there is more time
3. choose a random subset s
4. NEXT <- next(s)
5. if max{ W(v) | for each v in NEXT} < W(s): //s is a local maximum
5.1. if W(s) > best: best <- W(s) //if s is better then the previous result - store it.
5.2. go to 2. //restart the hill climbing from a different random point.
6. else:
6.1. s <- max { NEXT }
6.2. goto 4.
7. return best //when out of time, return the best solution found so far.
The above algorithm is anytime - meaning it will produce better results if given more time.
This can be treated as a linear assignment problem, which can be solved using an algorithm like the Hungarian algorithm. The algorithm tries to minimize the sum of costs, so just negate your weights, and use them as the costs. The assignment of rows to columns will give you the subset of points that you need. There are sparse variants for cases where not every (row,column) pair has an associated point, but you can also just use a large positive cost for these.
Well you can think of this as a binary constraint optimization problem, and there are various algorithms. The easiest algorithm for this problem is backtracking and arc propogation. However, it takes exponential time in the worst case. I am not sure if there are any specific algorithms to take advantage of the geometrical nature of the problem.
This can be solved by a pretty straight forward dynamic programming approach with a exponential time complexity
s = {A, B, C ...}
getMaxSum(s) = max( A.value + getMaxSum(compatibleSubSet(s, A)),
B.value + getMaxSum(compatibleSubSet(s, B)),
...)
where compatibleSubSet(s, A) gets the subset of s that does not overlap with A
To optimize it, you can memorize the result for each subset
Some way to do it:
Write a function that generates subsets ordered from the subset off maximum weight to the subset off minimum weight while ignoring the constraints.
Then call this function repeatedly until a subset that honors the constraints pops up.
In order to improve the performance, you can write a not so dumb generator function that for instance honors the not-on-the-same-row constraint but that ignores the not-on-the-same-column one.
The source data for the subject is an m-by-n binary matrix (only 0s and 1s are allowed).
m Rows represent observations, n columns - features. Some observations are marked as targets which need to be separated from the rest.
While it looks like a typical NN, SVM, etc problem, I don't need generalization. What I need is an efficient algorithm to find as many as possible combinations of columns (features) that completely separate targets from other observations, classify, that is.
For example:
f1 f2 f3
o1 1 1 0
t1 1 0 1
o2 0 1 1
Here {f1, f3} is an acceptable combo which separates target t1 from the rest (o1, o2) (btw, {f2} is NOT as by task definition a feature MUST be present in a target). In other words,
t1(f1) & t1(f3) = 1 and o1(f1) & o1(f3) = 0, o2(f1) & o2(f3) = 0
where '&' represents logical conjunction (AND).
The m is about 100,000, n is 1,000. Currently the data is packed into 128bit words along m and the search is optimized with sse4 and whatnot. Yet it takes way too long to obtain those feature combos.
After 2 billion calls to the tree descent routine it has covered about 15% of root nodes. And found about 8,000 combos which is a decent result for my particular application.
I use some empirical criteria to cut off less probable descent paths, not without limited success, but is there something radically better? Im pretty sure there gotta be?.. Any help, in whatever form, reference or suggestion, would be appreciated.
I believe the problem you describe is NP-Hard so you shouldn't expect to find the optimum solution in a reasonable time. I do not understand your current algorithm, but here are some suggestions on the top of my head:
1) Construct a decision tree. Label targets as A and non-targets as B and let the decision tree learn the categorization. At each node select the feature such that a function of P(target | feature) and P(target' | feature') is maximum. (i.e. as many targets as possible fall to positive side and as many non-targets as possible fall to negative side)
2) Use a greedy algorithm. Start from the empty set and at each time step add the feauture that kills the most non-target rows.
3) Use a randomized algorithm. Start from a small subset of positive features of some target, use the set as the seed for the greedy algorithm. Repeat many times. Pick the best solution. Greedy algorithm will be fast so it will be ok.
4) Use a genetic algorithm. Generate random seeds for the greedy algorithm as in 3 to generate good solutions and cross-product them (bitwise-and probably) to generate new candidates seeds. Remember the best solution. Keep good solutions as the current population. Repeat for many generations.
You will need to find the answer "how many of the given rows have the given feature f" fast so probably you'll need specialized data structures, perhaps using a BitArray for each feature.
I've stampled upon a curious problem.
I've got an unbounded chessboard, N knights starting locations and N target locations.
The task is to find minimal number of moves for all knights to reach all target locations.
I know that shortest path problem for a single knight can be solved using breadth-first search, but how can it be solved for multiple knights?
Sorry for my english, I use it seldom.
You can compute the cost matrix as suggested by Ricky using breadth first search. so now, cost[i][j] denotes the cost of choosing knight i to goto end location j. Then you can use the Hungarian algorithm to find the final answer, which can be computed in O(N^3) complexity.
I assume you know how to do it for one Knigt .
You can reformulate your problem as a linear program:
I will use the following notations :
We have N knights and N en locations.
xij = 1 if you chose knight i to go to location j and 0 otherwise
cij is the min length of moving knight i to location j
Then you have the following linear program :
variables:
xij for i j in [0,N]
Cost function :
C= SUM(cij.xij for (i,j) in [0,N]x[0,N])
constraints:
SUM(xij for j in [1,N]) = 1 //exactly one knigt goes from i to j
SUM(xij for i in [1,N] ) = 1
(The matrix (xij) is a stochastic matrix)
if X is the matrix of (xij) you have n! possible matrix. This problem can be NP-Hard (there is no easy solution to this system, solving the system is pretty similar than testing all possible solutions).
EDIT:
This problem is called the assignment problem and there exist multiple algorithms to solve it in polynomial time . (check out #purav answer for an example)
As mentionned by #Purav even though this kind of problems can be NP-hard, in this case it can be solve in O(n^3)
About the problem #j_random_hacker raised :
Problem
If a knight is at a endpoint, the next knights should not be able to
go through this endpoint. So the Cij might need to be updated after
each knight is moved.
Remarks :
1. multiple optimal paths :
As there is no constraint on the side of the chessboard (ilimited chessboard), the order in which you do your move for achiveing the shortest path is not relevant, so there is always a lot a different shortest path (I won't do the combinatorics here).
Example with 2 knights
Let say you have 2 K and 2 endpoints ('x'), the optimal path are drawned.
-x
|
|
x
|
K-- --K
you move the right K to the first point (1 move) the second cannot use the optimal path.
-x
|
|
K
|
K-- --:
But I can easily create a new optimal path, instead of moving 2 right 1 up then 2 up 1 right.
1 can move 2 up 1 right the 1 up 2 right (just inverse)
--K
|
-
| K
| |
: --:
and any combination of path works :
1 U 2 R then 2 U 1 R etc... as long as I keep the same number of move
UP LEFT DOWN and RIGHT and that they are valid.
2. order in which knights are moved :
The second thing is that I can chose the order of my move.
example:
with the previous example if I chose to start with the left knight and go to the upper endpoint, dont have anymore endpoint constraint.
-K
|
|
x
|
:-- --K
-K
|
|
K
|
:-- --:
With these 2 remarks it might be possible to prove that there is no situation in which the lower bound calculated is not optimal .
BFS can still work here. You need to adjust your states a bit, but it will still work:
let S be the set of possible states:
S={((x1,y1),(x2,y2),...,(xn,yn))|knight i is in (xi,yi)}
For each s in S, define:
Successors(s)={all possible states, moving 1 knight on the board}
Your target states are of course all permutations of your target points [you don't actually need to develop these permutations, just check if you reached a state where all the squares are "filled", which is simple to check]
start=(start_1,start_2,...,start_n) where start_i is the start location of knight i.
A run of BFS, from start [the initial position of each knight], is guaranteed to find a solution if one exists [because BFS is complete]. It is also guaranteed to be the shortest possible solution.
(*) Note that the case for single knight is a private instance of this solution, with n=1.
Though BFS will work, it will take a lot of time! the branch factor in here is 4n, so the algorithm will need to develop O((4n)^d) vertices, where n is the number of knights and d is the number of steps needed for a solution.
Possible optimizations:
Space: Note that because BFS uses a lot of memory [O((4n)^d)] you might
want to use Iterative Deepening DFS, which behaves like a BFS,
but consumes much less memory [O(nd)], but takes more time to run.
Time: To accelerate the solution search, you can use A Star with an
admissible heuristic function. It is also guarenteed to find a
solution if one exists, and also ensures the solution found is
optimal, and will probably [with good heuristic] need to develop
less vertices then BFS.
So, I've found the solution.
BFS won't work well on an unlimited chessboard. There is no point in using any shortest path algorithm -- number of knight's moves from location a to location b can be computed in O(1) time -- M. Deza, Dictionary of Distances, p. 251
http://www.scribd.com/doc/53001767/Dictionary-of-Distances-M-Deza-E-Deza-Elsevier-2006-WW
The assignment problem can be solved using mincost-maxflow algorithm (eg. Edmonds-Karp):
http://en.wikipedia.org/wiki/Edmonds%E2%80%93Karp_algorithm
Given an array of items, each of which has a value and cost, what's the best algorithm determine the items required to reach a minimum value at the minimum cost? eg:
Item: Value -> Cost
-------------------
A 20 -> 11
B 7 -> 5
C 1 -> 2
MinValue = 30
naive solution: A + B + C + C + C. Value: 30, Cost 22
best option: A + B + B. Value: 34, Cost 21
Note that the overall value:cost ratio at the end is irrelevant (A + A would give you the best value for money, but A + B + B is a cheaper option which hits the minimum value).
This is the knapsack problem. (That is, the decision version of this problem is the same as the decision version of the knapsack problem, although the optimization version of the knapsack problem is usually stated differently.) It is NP-hard (which means no algorithm is known that is polynomial in the "size" -- number of bits -- in the input). But if your numbers are small (the largest "value" in the input, say; the costs don't matter), then there is a simple dynamic programming solution.
Let best[v] be the minimum cost to get a value of (exactly) v. Then you can calculate the values best[] for all v, by (initializing all best[v] to infinity and):
best[0] = 0
best[v] = min_(items i){cost[i] + best[v-value[i]]}
Then look at best[v] for values upto the minimum you want plus the largest value; the smallest of those will give you the cost.
If you want the actual items (and not just the minimum cost), you can either maintain some extra data, or just look through the array of best[]s and infer from it.
This problem is known as integer linear programming. It's NP-hard.
However, for small problems like your example, it's trivial to make a quick few lines of code to simply brute force all the low combinations of purchase choices.
NP-harddoesn't mean impossible or even expensive, it means your problem becomes rapidly slower to solve with larger scale problems. In your case with just three items, you can solve this in mere microseconds.
For the exact question of what's the best algorithm in general.. there are entire textbooks on it. A good start is good old Wikipedia.
Edit This answer is redacted on account of being factually incorrect. Following the advice in this will only cause you harm.
This is not actually the knapsack problem, because it assumes that you cannot pack more items than there is space for in some container. In you case you want to find the cheapest combination that will fill up the space, allowing for the fact that overflow may occur.
My solution, which I don't know is the optimal but it should be pretty close, would be to compute for each item the cost benefit ratio, find the item with the highest cost benefit and fill the structure with this item until there isn't space for one more item. Then I would test to see if there was a combination with any of the other available items that could fill the available slot for less that the cost of one of the cheapest items and then if such a solution exist, use that combination otherwise use one more of the cheapest items.
Amenddum This may actually also be NP-complete, but I am not sure yet. Anyway for all practical purposes this variation should be much faster than the naive solution.